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.FOREWORD 



Legislators, governors, statewide coordinating and governing 
boards and the academic community generally, have major responsibilities 
in assuring that the best possible programs and the optimal array of 
programs are offered by institutions of higher learning. How these 
responsibilities are carried out varies greatly from state to state. 

Efforts to expand or retain programs are avid in some places 
and at times are irresistible. Competition for development of programs 
is intensive both within and among many institutions.' In funding 
institutions of higher learning, the state must hold them accountable 
to the public. 

The crucial and delicate issue is how colleges can maintain 
autonomy, which is imperative if they are to serve well in a free 
society, and on the other hand, how the state .can be assured that 
maximum value is obtained from its investment in programs of higher 
education. 

The study by Robert 0. Berdahl, Professor of Higher Education, 
State University of New York and Robert J. Barak, Director of Informa- 
tion and Research, Iowa Board of Regents, is an excellent analysis 
of how the states are addressing the review of programs. Both 
practical and theoretical perspectives are presented, and specific 
recommendations are tendered . 

The study has been done in response to interest expressed through- 
out the country on this topic. 

The Inservice Education Program of the Education Commission of 
the States and readers of this report are in the debt of the authors 
for analyzing well a difficult topic with considered, judgments. 

Appreciation goes also to Amy Plummer who helped in editing this, 
the second edition of the study, and to Adrienne Sack who typed the 
manuscript . 

Louis Rabineau, Director 
Inservice Education Program 
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I. INTRODUCTION 

The development of higher education in the United States has been primarily 
an incremental process. As new student populations and financial invest- 
ments vastly enlarged the undertaking, new programs, even new kinds of 
institutions, have been added on, constantly changing the shape and 
nature of the enterprise. Today, however, the process of change is not 
one of growth, rather it is one of contraction. Current and anticipated 
enrollment declines and the drying up of funding sources, spell an end to 
expansion. 

While some refer to the contemporary situation as a "steady state/\we 
concur with others who regard the situation as anything but" "steady."^ 
Lyman A. Glenny, for example, points to numerous factors that essentially 
suggest retrenchment: the reduction in categorical federal programs of 
institutional aid with a corresponding shift to student tuition assistance; 
the decline in birth and college attendance ra*es; the diminishing pro- 
portion of aid allocated to higher education in. state budgets; and finally, 
the ravages of inflation. 1 

Yet financial retrenchment need not result in the loss of viability or 
quality in higher education. And the process of contraction need not 
parallel the somewhat mindless incremental process of growth. As Warren 
Bennis has commented, this could be a period of "creative retrenchment."*' 
Ways must be found to enable higher education to remain viable and 
dynamic within the constraints of existing or diminishing ox, at the ve.y 
best, slowly increasing resources. 
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In this study we explore one such way for preserving the quality and 
responsiveness of higher education, the academic program review process. 
The state level program review process has developed in response to 
growing pressures for accountability, coordination and efficient use 
of resources and we believe it holds the promise of helping what could 
be a sorry situation. State agencies around the country, many themselves 
but recently emerging, are only now beginning to make serious use of 
the review process. Standing in their way are various difficulties 
e.g., institutional inertia and mistrust, the. threat of political inter- 
ference from the executive and legislative branches, and a set of 
procedural and technical hurdles that must be worked out to assure 
quality, fairness and cooperation. 

The goal of our study is to document and analyze the current state of 
the art of academic program review, and in our final chapter to recommend 
procedures for possible adoption by the states. Our study and recommen- 
dations are based on multi -state site visits, interviews add surveys. 
(The methodology we used is described in Appendix A.) 

A. Need for Institutional Action 

The most obvious candidates for preserving the quality and vitality of 
higher learning, are, of course, the colleges and universities themselves. 
Being closest to the problem and being most concerned about autonomy, 
the institutions might be expected to take innovative .and energetic 
steps to counteract the negative consequences of declining resources. 
The existing evidence, however, suggests that most colleges, universities 
and even multicampus systems have been something less than responsive; 

\ . 
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rarely have they seriously reordered priorities or undertaken any kind 
of extensive program evaluation. 

It is not that the colleges Ibid universities are unaware of the 
troubling times. In a recent Glenny study covering the years 1968 
through 1974 conducted for the Carnegie Council on Policy Studies in 

Higher Education, nearly 1,000 administrators expressed their belief 

/ 3 

that leveling enrollments and funding were damaging to program quality. 
Among the factors they attributed to the decline in quality were: 
reduction in course and program offerings; decrease in faculty; cuts 
in expenditures for equipment, books, related services and for course 
development, experimentation and program innovation. The administrators 
emphasized the difficulty they had in reallocating existing resources - 
Yet the study noted that only three percent of the schools represented 
had engaged in "extensive" graduate program elimination or consolidation. 
Another 27 percent reported that they had eliminated or consolidated "some" 
graduate' programs . 

Several other studies confirm the Carnegie findings. Fulkerson, in a 

report of the American Association of State Colleges and Universities, 

noted that frequently institutions have responded to financial exigency ' 

by- "... across-the-board cuts which remove the need for decision 

4 

making about who, or what positions, need to be cut." When this type 

of px-ogram reduction was, resorted to Fulkerson notes that: 

this method of resolving the financial problems produced 
a barrage of questions about the college or university's 
mission, questions which affect the long-term planning 
and future of the institutions, but which were ignored 
due to the immediacy of the impending fiscal reduction. 
Thus faculty and administrations needed time to assess the 
potential effects of their actions on the programs, the 
student and the educational goals. This time was not 
always available. 5 



This common tenth hour response to budgetary woes indicates that few 
institutions have begun to plan systematically for qhe likely program 
retrenchment ahead. 



In another study prepared for the National Board on Graduate Education 
(NBGE) , Breneman found that 

the financial stress and changing labor market conditions 
experienced by departments in the "arts and sciences" 
disciplines have not stimulated many major progr^ changes. 
Most departments visited in the course of this study seem 
to be following what^has been labeled elsewhere asTan 
"enclave" strategy, i.e., a conservative strategy designed 
to maintain the status quo. During the site visiti we 
observed little evidence, of leadership on the part lof 
graduate faculty or. administrators in pressing for a re- 
examination of the goals and purposes of the varioujs ' 
graduate programs. 6 

The final report of the NBGE stated that "if universities drift through 
the next four to five years in the hope that something wiUl brighten 
the picture, we foresee a wrenching and extremely damaging downward 
adjustment in the 1980 ? s that could be minimized by careful planning 



and action now. 
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The NBGE urged that 

faculty members, administrators , and graduate students 
in each university nrst discuss seriously the type o:: 
graduate programs that the institution can reasonably 
be expected to offer. This review should include a 
realistic assessment of the quality of current graduate 
offerings, an inventory of resources available (and 
lacking), prospects for graduate student support, i 
analysis of recent placements of new graduates, area^s) 
of special strength, and so forth. 8 



Lee and Bowen found in their study of multicampus systems that 

at least some campus administrators and perhaps some 
faculty woulJ rather not have to make decisions which 
appear required: it is easier to accept a program cut 
handed down from the central offices than to impose one 
on friends and colleagues. ^ 
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At multicampus systems progress toward comprehensive reviews of new 
and existing programs has also been slow. In their first study of 
multicampus systems in 1971, Lee and Bowen noted that reviews of new 
and existing academic programs were conducted in the context of 
expansion. 10 The emphasis was placed on discrete evaluations of 
proposed new programs on a campus -by-campus basis. Systemwide 
considerations were underplayed and reviews of existing programs were 
rare. By 1975, in their more recent study, Lee and Bowe'n found that 
"systemwide review of new academic programs has become more intensive, 
based on academic quality and campus mission in, six or more of the 
nine systems and on fiscal criteria in eight." Moreover, seven of 
the nine systems studied had instituted procedures for periodic review 
of existing graduate and professional programs. 11 But the authors 
found a wide divergence in the quality and comprehensiveness of these 
systemwide reviews, and the effectiveness of current reviews of new 
and existing academic programs in most multicanTpus systems is certainly 
questionable . 

T. Edward Hollander, Chancellor of Higher Education in New Jersey 

has written that: 

Shifting demographic and economic trends may result in 1 
structural and programmatic changes that will seriously 
test higher education. These undesirable outcomes are 
possible if we fail to understand and plan for exogenous 
changes that will influence the scope and mission of 
our enterprise. * On the other hand, we can try to 
understand and shape our future rather than be shaped " 
by it. (Hollander, 1975)12 



B. Need for Obj e ctivity 

Even if administrators and faculty were better able to take a hard look 

at tHeir programs, both new and existing, many observers believe their 

inherent self-interest robs them of- the objectivity necessary for an 

effective evaluation of programs. Professors Kaplan, Whitehouse and 

Williams of the University of Michigan have stated that M as far as 

possible evaluation should be made by objective, impersonal processes 

or by experts with no self-interest in the institutions being evaluated 

13 

or in the results of the evaluation " 

Dressel et alo have also noted that even ,f the desire of the institution 
to introduce a new program . . . introduces a bias in the data selected 
for inclusion or omission in the request and the way in which the data 
included are exhibited and interpreted . . . enrollment projections . . 
are likely to be optimistic, and the costs are likely to be under-* 
estimated. ^ 

C. Need For Aaaoimtability 

The disillusionment of the body politic with postsecondary education has 

<\ , 

resulted in a questioning of many current practices and in a demand for 

greater efficiencies. Howard Bowen has suggested that these demands fof 
accountability "reflect in part a failure of confidence in many of our 
institutions and in part frustration over rapidly increasing costs."*^ 
Many in higher education feel that such demands' threaten institutional 
autonomy and vigorously resist external pressures for greater accounta- 
bility. 

But such automatic responses frequently ignore the relative nature ,of 

\ • . '■• ■ ' ,, 



institutional autonomy. Kenneth Mortimer has noted that the "real 
-issue with respect to institutional autonomy and accountability is not 
whether there will be intervention by the state but whether the 
inevitable demands/for increased accountability will be confined to 
the proper topic and expressed through a mechanism sensitive to both 
public and institutional interests ."*^ ^ 

The proper balance between external and institutional interests in 
the program review. process is difficult to define. Even the prestigious 
Carnegie Commission on Higher Education inits report on state-higher 
education relations^ failed to define explicitly the relative respon- 
sibilities appropriate to academic program review for existing programs. 
It might be inferred, however, that the Carnegie Commission^ recommen- 
dations for state influence and even control over "major new endeavors" 
and "effective use of resources," implies a cor commitant responsibility 

: to undertake periodic reviews of new programs to ascertain whether they 

-./ft ■ 18- 
are being carried out consistent with the effective ^'e of resources. 

The search for a "sensitive mechanism", for reviewing academic programs 
is currently being made by diverse agehts (some of questionable sensitivity) 
including legislative program auditors, A recent .series of articles in 
Higher Education in New England indicates that legislators are becoming 
dissatisfied with merely inspecting budgets or passing laws. They are 
now wanting to know the impacts of their legislative measures'. As a 
result, legislators in some 28 states have established post-audit over- 
sight 'activities. While the effect of these activities on higher 

* 19 

education has thus far been limited, the implications should be obvious. 
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Glenny et al. in a study of budgeting in 17 states found that 14 have 

separate executive or legislative staffs that perform policy-oriented 

performance evaluations, as distinct from routine fiscal audits. In 

most instances the authors found that the separate audit staffs had 

made performance studies of colleges c-ud universities and noted that 

the staff's role "in budgeting for higher education will probably 

20 

grow increasingly important." The table below from the Glenny et al. 
study shows the "Performance Audit Staff Location and Studies of Higher 
Education," 



Performrnce Audit Location and 
Studies of Higher Education 





Location of Separate Audit Staff 




Reports to 
Governor 


Reports to 
Legislature 


Reports to - 
Independent 
Agency 


Total 


Audit Staff has 
made Studies of 
Higher Education 
(11 staffs in 9 
states) 


2 


9 




11 


Audit Staff has not^ 
made Studies of 
Higher Education 
(6 staffs in 6 
states) 


- 2 


3 


1 


c 
\j 


Total (17 staffs 
in 14 states) 

SmiTPP * I.vman A C 1 onmi- 


4 


12 


1 


17 



Source: Lyman A- Glenny et al., State Budgeting for Higher Education : 
Data Digest ,, (Berkeley: Center for Research and Development 
in Higher Education, 1975), p. 16. 



A forthcoming publication (Folger et al., 1977) provides a further in- 
depth description and analysis of stable-level performance audit and review 

21 " 

in higher education. 

The need for action, objectivity and accountability all indicate the 
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necessity for some kind of extra-institutional review. We believe the 
state higher education agency is the appropriate body to take on this 
responsibility. 

In a previous publication, Berdahl noted that 

Most scholars have concluded. . . that neither the organs 
of state government nor the institutions of higher education 
are capable of conducting the finely balanced assessments 
involved in program review, the former because the issues 
are too complex for nonprofessionals to handle and the 
latter because their own self-interest often inhibits 
their objectivity. Thus, coordinating agencies — 
because they combine a statewide perspective with a 
specialized knowledge of 'higher education — have 
increasingly been called upon to play a central role in 
thiese decisions. (Berdahl, 1971)22 

' It is perhaps axiomatic that few issues are more sensitive to higher 

educators than those involved in state-level program review and approval 

The idea that "outsiders," "state bureaucrats /V "representative's of a' 

political environment," or even a "governing board staff" might meddle 

in institutional academic affairs is probably more threatening than all 

other administrative and coordinative aspects of statewide coordination. 

The principle of "peer evaluation and review" in academe appears to be 

as well established as the skepticism towards program .review performed 

23 

by anyone outside academe. > 

Despite this entrenched institutional preference for self-determination, 
the role of statewide coordinating/governing agencies in the program 
review and approval process has been growing steadily. The development 
of the program^ review function has paralleled the development of the 
agencies themselves. 
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D. Growth of State Agency Reviews 

The number of state level coordinating/ governing agencies for higher 
education and the role and scope of their responsibiJ ities have 
steadily increased. In several states the early coordinating agencies 
were largely dependent upon the voluntary efforts of the individual 
institutions. Subsequently this form of coordination proved un- 
satisfactory and new agencies were established with more regulatory 
responsibilities over higher education. More recently, in North 
Carolina, Wisconsin and Utah, coordinating agencies were abandoned; 
consolidated governing boards replaced the institutions 1 or systems 1 
boards of trustees. The new consolidated boards thus had virtually 
complete governing and coordinating powers over public higher education 

Usually included among the powers of the state-level agencies has been 
the responsibility for program review. The first reviews were 
primarily for new programs but more recently have been expanded to 
include existing programs as well. Like the development of agencies, 
themselves, the program review function has emerged from relative 
obscurity to an important role in the- coordination and planning of 
higher education. While there were some early examples of governing/ 
coordinating boards playing a decisive role sin curricula matters, even 

to the point of approving textbooks, these were clearly the exceptions 

24 ■ r ' 

rather than the rule. Most early coordinating agencies did not 

involve themselves in academic programs. 

Beginning in the 1950's, however, some degree of . program review was 
exercised>by a few coordinating agencies primarily in relation to the 

> J Is 
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budgetary process. The classic example of this is the Board of Education 

Finance in New Mexico which influenced a control over programs through 

25 

its budgeting powers. The New Mexico Board influenced program 
decisions through such techniques as requesting and sometimes publishing 
data on class size, teaching loads, cost per student -credit hours and 
number of graduates per year. In addition, it asked that all new programs 
and services be listed as sepiarate budget items and be fully justified. 
According to Berdahl the board then ". . . pushed steadily to cut back 
on redundant courses and degree programs by pegging faculty salary 
increases to increases in class size,! by refusing to budget naw faculty 
positions when data indicated that z : department lacked self-restraint. . 
> - 

As budgeting procedures became more refined throughout the country with 

the introduction of uniform accounting practices and information T 

systems in the early 1960 ! s, it became clear that more exacting program 

review procedures were also needed. Program Planning and Budgeting 

Systems (PPBS), for example, were developed with the long-range goal 

of having M . . . multiyear estimates of both costs and benefits of 

27 

every proposed .program and its major alternatives." (To date the 

28 J 

efforts at developing PPBS have. fallen short of this goal.) 
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Both budget review and program review were alscr soon recognized as 
sharing an important relationship with long-range, statewide planning. 
As the emphasis on planning grew, the program review function became 

an . . important power in its Implementation and orderly development. 

• . >•■ 

Planning during the 1960 f s wats primarily concerned, with the extension 
of educational opportunities to a larger and more diverse student 
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population. Consequently, program review largely involved the 

establishment of new institutions, new campuses and new programs- The 

nature of the relationship between program review and planning during 

that period has been described by Glenny et al., who asserted that 

statewide planning, if it has been done and done well, could have 

established institutional missions within which proposed new programs 

31 

could be evaluated. However, the authors regarded program review 
as .indispensible, whether or not long-range planning had occurred. 32 
During the 1960's the review of existing programs occurred only 
sporadically. 

Glenny et al. points out- that little debate now exists .on the necessity 

of a centralized review of high cost, graduate programs or of a 

- \ ■• • • ' . \ ■■ 

centralized determination of priorities in the development of. new 

programs or campuses. The authors note, however, that enthusiasm 

decreases markedly for 'the review of undergraduate programs and there 

is considerable apprehension towards agency powers' over course approval 

and reallocation or elimination of existing programs. The elimination 

of existing programs is extremely sensitive since it may involve changes 

in staff, disruption of student expectations and the reallocation of 

resources. 

Nevertheless, the recent Education Commission of the States (ECS) Task' 
Force on Coordination, Governance and Structure recommended that the 
scope of program review and evaluation be expanded to include reallo- 
cation, reorganization or even discontinuance of units as defined. 
This expansion in the scope of program review was deemed necessary by 
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the need for retrenchment, and reorganization in response to decreasing 
34 

enrollments. However, the ECS task force also recommended that new 
courses be evaluated only at the institutional level. 

A comprehensive assessment of state-level program review functions 1 

has not been undertaken", although the disagreement has' appeared with 

Glenny's 1971 assessment: 

An overall. assessment of the program review function \ 
is that it has been moderately successful . in controlling \ 
new program expansion and in setting diverse functions 
among institutions. Agencies have been far less 
successful in eliminating existing programs or N in 
maintaining differential functions. 35 

E. State Agency Structure and Responsibilities for Program Review ' ■ 

The literature on statewide coordinating of higher education generally 

notes three different types of coordinating agencies. These ty£es are: 

I. Voluntary agenc.ies in which coordination is performed 
by the institutions themselves operating with' some 
degree of formality. .. 



II. Coordinating agencies established by statute ,but not 

superseding institutional or segmental governing boards. 
The statutory coordinating Agencies may be further 
divided, into advisory and regulatory types. Advisory 
coordinating agencies essentially have the power only 
to ''recommend 1 ' or "advise" whereas boards with more 
regulatory powers can actually require institution^ 
to comply with regulations within the agency's powers 
of jurisdiction. 

III. Governing agenciesjfunction either as the governing 
/ body for the only public senior institution in the 
state or as a consolidated governing board for 
K% ,k> multiple institutions, wich no loc?A or >segmental 
governing bodies. 36 ^ . 

These three types/of agencies differ considerably in their, respective 

powers over academic program review. As Berdahl has pointed out: 
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In theory all consolidated governing boards would have... 
such program review powers: What the board giveth, it can 
taketh away. Coordinating boards on the other hand vary 
widely in their powers to review programs, ranging all ' 
the way from no mention at all of the issue in the 
enabling legislation, to granting power only to 
recommend . . , to authorizing final powers in certain 
fields cr for certain types of institutions, to granting 
blanket regulatory power's over all public higher educa- 
tion, to even folding in private sector programs in 
New York. 37 



Boards also vary widely in the actual exercise of their powers. Some 
consolidated governing boards play a very limited role in program 
review while a few coordinating agencies with little or not statutory 
responsibility for program review have even reviewed existing programs 
and have been relatively successful in terminating some questionable 
programs.. While extremely rare, some voluntary coordinating groups 
have been successful in preventing program duplication. In Iowa, for 
example, a voluntary coordinating body was instrumental in achieving 
substantial modifications in a new program proposed by a private 
institution. 



.Coordinating type boards differ considerably in both authority for 
program review and in their use of that authority. It has been widely 
recognized that legal authority for review by an, agency may not, for 
numerous reasons, be exercised; on the other hand, authority which may 
not exist in law may be exercised by other means. 38 An example of the 
latter was the situatiqn^in New Mexico mentioned; above where program 
review powers emerged de facto frcm the exercise of budgetary powers. 



The growth instate-level agency program review .responsibility is 

.39 40 

"illustrated in Table 1 based on data from 1960 and 197S surveys-. 
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Table 1 



Growth in State-Level Program 
Review Responsibility Since 1960 



Criteria 


1960* . " 


1975** 


Governing 
Agencies 

# % 


Coordinating 

rig cuticb 

*>*•-. % 


Governing 
agencies 

# % 


Coordinating ~~ 
Agencies 
* % 


Number of 
Agencies 


16 100 


8 100 


' 19 100 


.28 100 


Program Approval 
Authority 


16 100 


3 38 


19 100 


20 72 


Program Review & 

Recommendation 

Only 


0 0 . 


4 50 


0 0 


8 28 


No Program 
Review 


0 0 


1 0 , 


0 0 


0 0 


*Source: : S. V. Martorana' and E. V. Hollis, 


State Boards Responsible for 



Higher Education , U.S. Office of Education, I960.-. 
**Source: Higher Education in the States, Education Commission of the 



States, "Survey of the Structure of State Coordinating or 
Governing , Boards and Public Institutional and Multicampus 
Governing Boards of Postsecondary Education As of January 
1, 1975." Volume 4, No. 10, pp. 297-352. 

'»■ ■ \ 

In "I960 there were,,sixteen state-level governing boards, and in 1975 
there were nineteen governing boards, all of which had potential program 
review authority. 1 In 1960 there were eight state-level coordinating 
agencies, three with program approval authority and four with authority 
"to recommend." In' 1975 tfrere were twenty-eight coordinating agencies 
. of which twenty had program review authority and eight had authority 
to recommend program approval. 

-£V Relation of Program Review to Budget Review ^.and Planning . 
\Whrle program review is now_viewed as a process in its own right, it has 
an^im^ortant and growing relationship to both budgeting and planning. 

Neither p^rqgram review, nor budgeting, nor planning, can occur in 

, j \ • . 
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isolation if institutions are to respond, creatively to the possibility 
or actuality of retrenchment. Obviously, programs require appropriate 
financing if they are to be meaningful; and if necessary financing is 
to be available, appropriate institutional, segmental and state planning 
needs to take place. 

The example of New Mexico's early approach to program review demonstrates 
how budgetary responsibilities are often related to program review. In 
some other states, approval of a new program is often contingent upon 
termination of an. existing program or upon the specific appropriation of 
funds for the program. 



The previsions for new program approval in a number of- states also 
require that the' program be integrated into the state master planning 

A. 

process prior tor state-level approval. " 

The ultimate attempt to integrate planning, programming and budgeting^ 
has -been -the so-called Planning, Programming and Budgeting-System (PPBS) . 
But PPBS has proven to be of limited value when applied to higher edu- 
cation. Its most serious problem is related to time horizons. Balderston 
has noted that: 

One basic idea in PPBS is to specify what is to be 
accomplished in each program in each future year and 
then to calculate how many resources will be needed for 
the program and to ^ specify when these resources have to 
-be acquired and at -what cost in order to meqt-the schedule^ 
for the .program. This sys*&m forecasts the cost of the 
program over a series of years and sets forth the timing 
of these costs, thereby avoiding seemingly innocuous . 
first- year commitments to' programs that eventually turn 
out to be far more costly, than originally contemplated. 
It is still quite possible to make a poor choice of 
program or to underestimate its costs,, and PPBS did not 
altogether solve the problem of mistaken and misunderstood 



commitments, even in the Department of Defense, But it 
was certainly a far more sensible conceptual approach 
than the political jockeying that got programs started 
on false promises and then built them into unstoppable 
and large claims later on. 

Universities found that, although they tried to show 
the multiyear budgetary implications of programs, their 
funding sources were unwilling or unable to look beyond 
^yery short commitments typically, the single budget 
year. Administrators were very much aware that most 
of what they were trying to sustain in existing programs 
or to initiate in new ones had implications for costs 
and horizons for results that stretched far beyond the 
immediate arguments of the up-coming fiscal year. But 
they have generally not been able to deal with the 
multiyear horizon except in terms of very contingent 
internal planning*or by announcing noble hopes whose - 
realisation would be heavily dependent on future 
decisions of outside funding agencies. The most enduring 
legacy of the program budgeting experience of universities 
has been the development of a much more .sophisticated 
analytic spirit, both within the university and in state 
and federal agencies. 41 
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II. ANALYSIS OF NEW PROGRAM REVIEWS 

Our analysis will consist of three parts: a discussion of what "programs" 
are reviewed; the criteria used in reviewing new programs; and the process 
or mechanism of review. It became, quite evident early in^our state 
visits, that the process or mechanism of review was the most crucial 
factor in effective reviews. When participants viewed the process as 
flexible, and fair, they then tended to perceive the overall review 
more favorably despite any identified deficiencies in the selection of 
programs reviewed or the criteria used to review them. The latter two, 
if deficient, were often viewed as susceptible to later improvement 
through a fair and flexible process. Consequently, primary emphasis 
in this analysis will be placed on the process of program review. 

=4. Programs to be Reviewed 
Which programs were subject to" 'state level review Varied considerably 

"among "the"" states"" Some state higher "educa^ ~ 1 

major new graduate or doctoral programs involving legislative requests 
for new-funding. At the other extreme some state agencies review -.all 
new academic programs, all changes in degree designation, all new tracks 
(toajors and minors within a degree designation) regardless of whether. 
any new costs are involved. A few agencies even review proposed course 
changes . 

The lack of a specifi Redefinition of programs subject to review was a 
frequently r^orted sourc^of major problems. The definition of program 
used by a particular coordinating agency needs. to be specific enough 
not to be subject to institutional variation. -The distinctions between 




program, major, area of emphasis, concentration, option, and specialty- 
were frequently found to be so vague that institutions could/ if they 
wanted to, reclassify a pre/gram to avoid a review. In Montana, for 
example, institutions offered, through continuing education or ^summer 
session, courses which they would not" otherwise have been eligible to 
offer (a new policy now prevents this). In South Carolina, the definition 

"major" had to/ be tightened due to the penchants of several institu- 
tions for labeli/ng groups of courses concentrations and insisting they 
were not new programs. > 

Andrew J. Rudnick has proposed the following definition of "program" 

fox use in proposed new legislation in Alabama: 

(1) Any instruction or series of courses not previously ir the 
• cur?icul?un and arranged in, a scope and sequence leading to 
a degree at any level, or to a credit certificate^ and y 
including new majors in any degree program already 
established. 

(2) L.Programs in research, public service, or any combination . ' 

of these with instructional programs "l?ut~"only wHen~sucTv " ~ 
programs depend in significant part or in full upon 
direct legislative appropriation for support. 

(3) The initiation of programs, either existing or new, in 
teaching, research, or public service in new off-campus 
locations, 

(4) The creation or establishment of a new.public institution ,, 
of higher* education, either by the creation of an entirely 
new entity or by the merger with or consolidation of... 

■ . existing entities, public' or private, or; by the transfer or 
assignment of any teaching, research, or public service 
programs by and among existing institutions or systems, 
• public or private. 42 



One statewide coordinator reported that his Estate had solved' this, problem 
by adopting the principle that ."...if a program (By whatever name) is to 



be advertised in any media as a separate identifiable . entity, it is ", •:> 
then subject to the review mechanism. . . and if in doubt it should be subject 
to the review process." 

An important distinction in the. definition of programs to be reviewed ■' 
proved, to be that between degree programs and tracks or agencies 
reviewing only full degTee programs have frequently encountered the 
problem of m&ny institutions asserting that the new program n won> f t o 

cost anything because the courses are already being taught and the faculty. 

... * 1 * 

and other resources are already on board." In some states institutions 

have also developed tracks as options under an approved program rather 

than designating them as a new program, thus, they avoided the concomitant 

approval procedures. The review of courses is one way to approach the 

problem. However, the review of courses, especially in a large £tate, 

is deemed unmanageable and could more easily interject the coordinating' 

agency int<^ infringements on academic freedom. Course control, . as 

proposed by one agency staffer l?elow, does have some possibilities: 

If the rationale. Hot controlling progress is to attempt to 
maintain average class size at an efficient level by * " 
restricting curriculum proliferation it might be more 
effective to control the number of courses' (or course 
credit hours.) which an institution may offer, in addition 
i; to the number of courses. Presumably the authorized 
number of courses woiild reiate fo the total student 
body size, and to authorized degree programs. v If a 
number could be established for the maximum number of 
courses to be offered at. an institution the institution 
would then be permitted to add additional courses up 
to that limit.. In order to create new courses after 
the limit is reached, the institution would be required^ 
to delete an existing course f or obtain. authority .to 
increase the number of courses offered. 

The number of authorized courses could be controlled by 
level and discipline as well as by total , niunber. If 
this were done, the process^ for developing the number 



of courses jhight be the counting of courses required for 
various authorized degree programs and the identification 
of courses shared by several degree programs. Authorization 
of new programs or additional courses could then be coupled. 
Proposals for new programs would specify nuriiber of new 
courses to be added and identification of existing .courses 
to be used. 

Monitoring and enforcement could be handled by maintaining 
a 1 file of authorized courses and matching with data on the 
Student Data Course File, Any course on the Student Data 
CouTse File and not on the currently approved list would 
not generate fundable credit hours. The List of currently 
authorized courses could be modified annually by the 
university provided that the number of courses at each 
level in each discipline did not exceed the authorized 
number. 



Perhaps more effective than course control has been the review of tracks, 

• * 

majors and even minors within a degree designation. The review of 
tracks may eniable the coordinating agency to review an area of speciali- 
zation before significant resources have been committed'. Program inven- 

tories in" Florida and Kansas now include tracks or subspecialties. 

p 

The scope of programs reviewed by the state higher education -agencies 
seemed to be a result of either a statutory or a voluntary delineation^ 
of programs to be reviewed. The statutory aspects of program review 
may be quite explicit such as those in Indiana quoted here: 

\ 

Public Law 326. Sec. 8^30 Programs.. To approve or disapprove 
the establishment of any\iew branches, regional or other 
campuses or extension .centers, or of any new college or 
school, V or the offering of any additional graduate degree. 
The Commission may review new associate and baccalaureate 
degree programs in any academic area by any state educa- , 
tional institution with the power to approve or disapprove. 

Statutory responsibilities may also be less explicit such as those in 

Iowa which confer upon the Board of Regents broad discretionary po^er 

5 
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to govern the state universities. The interpretation of statutes in 
this area is not always clear and in at least one state (Michigan) the 
coordinating agency's authority to grant prior approval of new programs .. 
at public baccalaureate institutions has been legally challenged,* 

It 'should be noted that the mere possession of statutory authority by 
a state agency to review academic programs does not necessarily mean 
that an agency does in fact review programs, nor is' it necessarily an 
indication of the extensiveness of that process. Nor does the mere 
absence of explicit statutory responsibility for program review 
necessarily inhibit an agency from reviewing, programs . Some of the 
most vigorous reviews are conducted by state agencies without specific 
statutory responsibility for program review. 

In some cases agencies have voluntarily limited the scope of prograin review 

in order to use t he ir' avail able staff resources most effectively .or - 

in order to concentrate efforts on areas of special concern such as 
graduate programs. 

In a number of states, agencies deliberately limited the overall, program 
review' effort or slowly expanded it in order not to upset the delicate 
balance between the institutions' strong desire for institutional^ 
autonomy especially in the academic affairs and state officials' desire 

*In this instance the State Supreme Court ruled' in essence that the 
state constitution provides, autonomy to the institutions in terms 
of their internal operations, that the role of the State Board of 
Education is advisory in nature on program review matters. However, 
the Court granted specific recognition to the State Board's planning 
and coordinating functions' by mandating that the institutions are 
required to provide information on new programs and financial re- 
quirements to the State Board in order for the Board to carry out : 
its advisory role. Constitutional changes have since been proposed 
that would increase the state agency's review authority. 
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for evidence of greater efficiency : in education. Below is a partial- 
list of the many different ways in which limitations have been made t 
the. kinds of new programs reviewed by many state agencies : 

1- B ; y program level (or general category M degree programs") 
a. lower division - (community/ junior college) 
upper division 

c. Masters 1 ~ 

p or Graduate 

d. Docto ral J \j 

l=B5S ^0' e. Professional 

2. By funding level or source 

a. . fixed amount 

b. need for state legislative appropriations 

c. type of external funding source 

3. By length of* program 

a. financial commitment, per number of years 

b. length of program duration 

4. By organizational level/unit 

a. instruction (college, school, institution, department) 

b. research 

c. administration 

d. public service " 

5. Changes of existing programs 

a. track . 

b. improve i ■ 

c. expanded 

d. terminated 
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6- By sector 

a. public 

b. private 

c. proprietary 

7. By certification 

8, By accreditation 

9; Credit and noncredit 



When to review a new program is another touchy issue. The California 

state agency has recently determined to consider new programs projected 

for initiation two to five years in the future, instead of reviewing 

and commenting on them only shortly prior to their implementation 

date. This approach requires a concomitant .emphasis on longer-range 

academic' planning, an emphasis which wa$, found wanting in some of the 

states visited. The California Commission staff provided the following 

description of the way in which this longer-range review will be done: 

In conjunction with an intersegmental council, the Commission 
has developed a schedule for the annual submission of long- f 
range plans to the Commission, and a process for determining 
progTam areas in which an excess of programs may now or soon 
exist, or which may require a special study. Among the 
j criteria for identifying such program areas are tho number 
and location of existing and proposed programs; the employ- 
ment market; the cost of the program; and student demand. 
The identification or certain program areas is intended 
to discourage the institution frc*n proceeding with -their 
plans for a new program in that area. If the college 
proceeds anyway, or if it proposes a program not appearing 
previously on its five-year plan, the Commission will 
review that proposal with special emphasis on the circum- 
stances cited in its justification, . K 

It is probably, too early to effectively assess this process in California 
since.it has only recently been implemented. A similar plan, is being 
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developed in Kentucky, This general approach should mitigate the 
problems of large numbers of individual program requests, their fre- 
quently late submission, and the potential . overlapping of the review 
^function as performed by institutional, segmental and coordinating 
'agency staffs. On the other hand this- approach may have the t ^advantage 
of restricting institutional responsiveness by preventing the swift 
development of programs for which there may be a unique and even urgent 
need. 

B. Criteria Used for Program Review 

Almost all of those we interviewed expressed concern about the criteria 
used to review new programs. We heard distress about the "quantitative" 
nature of most of the criteria and the relatively slight concern with 
"qualitative" criteria. Faculty were particularly critical on this point 
even though they generally admitted; that they had no suggestions on how 
to make the criteria more qualitatively oriented. The. quality of a 
program, they felt, could best be judged by peers within a given academic 
discipline. 

The biggest problem with "quantitative" measures was the fear that they 
are or could be used arbitrarily across institutions and disciplines 
without taking into consideration the many unique differences among 
types of institutions and disciplines. While some individuals totally 
ruled out the use of quantitative data for reviewing programs, most 
indicated that they would have less concern about their use if they were 
but one of several kinds of criteria employed in evaluating a new program: 
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The qualifications of the staffs of the state-level agencies doing the 
reviews was a related concern. Most state level §taff were categorized 
by institutional- representatives as lacking "academic experience 11 
which they invariably defined as ,f X" number of years in the classroom. 
Academics feared that such personnel, lacking the desired experience, 
would rely too heavily and r too inflexibly on quantitative measures. 
Even if thecstate-level personnel had creditable academic experience* 
their relative remoteness from the campus was seen. as possibly diluting 
their perception of academic concerns. 

Figure I provides a simplified profile of the seven major factors 
commonly included for new program review in the various states. 
The seven factors are: (1) program description; (2) purposes and 
objectives; (3) need analysis; (4) cost analysis; (5) resource analysis; 
(6) program accreditation; and (7) availability of adequate student 
financial aid. (The ..degree of specificity of the required response to 
each component varies .'considerably between the states.) 

1. Program Description. T]ie first, required, factor for a new program 
review is a description of the program being proposed. This could range 
from a simple catalog-type statement to a comprehensive, well-documented 
xeptfrt including the entire proposed and related curriculum, prerequisites, 
credits, method of instruction, degree (s) to be granted and courses 
taught by related departments. The program description also establishes 
the context within which the other factors may be considered. 

2. Purposes and Objectives. While the degree of detail required 
in this statement varies, it generally consists of what the proposed 
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FIGURE 1 (continued} 
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"x" indicates those categories which a specific state utilizes in its program review process" based upon information L 
■■ obtained from either specifically expressed or implied that agency. 7 

• Q ' 

(1) New programs have provisional s'fatus first full cycle. 

(2) . lias a. two phase process of program review. Phase I is similar to a letter of intent but more comprehensive. Phase 

II is considerably more detailed. 

(3) Requires a ranking of state and institutional priorities. 

(4) Separate procedure for new health programs. , > . • 



FIGURE 1 (continued) 



(5) The Minnesota review process is currently being revised to include increased detail and clarification. Present ■ 
■ criteria include: mission, .unwarranted duplication, cost-benefit and need for program.' 

(6) ' Currently being developed. • . 

(7) Criteria listed apply generally to all. programs; however, there are separate criteria for each degree level (i.e., 
B, M, D) 

(8) ' Doctoral programs must submit more extensive information. State college programs requiring additional fiscal 

resources need both council and legislative review. 

(9) Vermont is in the process of developing program review procedures. 

(10) Has a three stage process: intent to plan, program approval, operational approval. 

(11) These criteria are optional and apply to graduate programs only. /J 

(12) Includes private institutions (in some instances .this is voluntary on the part of private institutions). 

(13) University, of Nebraska only. 

(14) West, Virginia uses, a procedure which is difficult to represent in this table (see description in section on 
existing program.) 

(15) North Dakota has a procedure for review but specific criteria were not provided in response to survey. 

(16) Public institutions must seek approval for planning and licensure, private institutions only the latter. 

(17) Includes public and private institutions. 

(18) ■ Program review and long- range planning are integrated. into a process in which programs. projected for initiation 

two to five years in the future are considered annually., 
(1.9) The criteria for review of new programs are currently being reassessed. 
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program is intended to achieve and how it interrelates with the institu- 
tional (or other unit) mission and the state master plan (if one exists) . 

In North Carolina the University System, for example, is developing per- 9 • 
formance goals for proposed programs in terms of projected enrollment, 
anticipated accreditation, modification of facilities and faculty develop- 
ment. These performance goals will be a basis for periodical review of 
programs after establishment. 

3. Need Analysis . This section in most states is a justification 
for the proposed program generally concerned with: the need for the 
program at the institution and in the geographical area it serves (i tv e., 
state, regional, national or international); the student demand for the 
program; the projected enrollment; the relationship to manpower needs; 
the opportunities for employment; the likelihood of unnecessary program 
duplication within the state and "ttempts to establish cooperative 
arrangements with other institutions. 

Although such justifications for a proposed program are clearly important 
there is some agreement that many factors are?* just based on "quesstimates . " 
For example, manpower projections are notoriously unreliable. Often the 
reviewers are confronted with different and even conflicting estimates 
of need and must judiciously exercise their professional judgment. 

The determination of "unnecessary duplication 1 ' is also frequently a judg- 
mental matter. Some duplication within a given state is viewed as 
necessary -and highly desirable while other program duplications may be easily 
recognized as unnecessary. Unnecessary duplication of the programs between 
these two extremes is much more difficult to determine. 
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Martorana and Kuhns have developed a methodology for analysis of 
duplicate program offerings on a statewide basis. The principal objective 
of their research project was the design and implementation of a model 
for assessing possible program duplication. The project focused on the 
need for, availability, supply* and possible oversupply of degree programs 
at five levels — associate, baccalaureate, masters, first professional, 
doctoral — in all types of public, private and proprietary institutions 
chartered to grant degrees in a large eastern state. A basic assumption 
was that pairs of program^ with the same Higher Education General 
Information Survey (HEGIS) classification, at the same degree level 
must be compared in order to determine the extent of program duplication. 
For each degree program a "unit record" containing a number of data items 
(program descriptors) was established. Using these unit records, 
"programming profiles" were developed for types of institutions, degree 
levels, and educational planning regions. Included in the programs' unit 
records were indices of geographic co-existence or duplication, availa- 
bility, graduate production, similarity, institutional need or dependence 

43 

on the program, and student and manpower demand for the program. 

A number of states either have, or are in the process of developing, 
criteria which would examine the extent to which cooperative programs 
between institutions and within consortia arrangements have been explored 
prior to the introduction of new programs. Such arrangements are viewed 
as important alternatives to a single institution developing an entire 
new program unilaterally. A statewide coordinator in the midwest 
indicated that "increasingly it will be necessary to develop certain 
programs on a statewide rather than just an institutional basis." 
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4". Cost Analysis, This sectioh establishes the cost (e.g., direct, 
indirect, incremental and reallocated) of the proposed program for a 
given time usually ranging from two to six years. The factors generally 
involved in instructional program costs include the amount required for 
direct teaching, for instruction-related activities, library, administra- 
tive, research and service costs when appropriate. 

*? * 

Also frequently considered in cost analyses are the sources of funds for 
the proposed program. Many states differentiate the extensiveness of the 
review by the source of funds. State funded programs have generally 
received the most extensive reviews. Also, if state funds are needed 
for a program, the proposal , even if approved with respect to all other 
factors, may, as in Oklahoma and Arkansas, be dependent upon the subsequent 
appropriation of state funds for the program. ^Programs financed fully by 
nonstate funds used to escape close fiscal scrutiny, however, because many 
states have recently been asked to pick up the tab for programs formerly 
supported by nonstate, usually federal funds, the costs examined now 
include all source funds. , ~ 

An important problem associated with this cost analysis is the frequent 
institutional assertion that the proposed programs "won't cost anything 11 
since the faculty and other resources, are already, in place. In one sense 
this appears to be true: over the /ears „ faculty with special expertise 
were added; courses reflecting these faculty interests were added; 
necessary equipment and resources to supplement the courses were acquired; 
and, finally, minors in the area were offered. As a consequence, the 
formal establishment of the new program is like an afterthought . In 
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another sense, however, the legitimization of the program does cost 
something — the faculty must be paid, the resources maintained, the 
overhead accounted foT — and many states fpel that these ' <fbsts should 
be identified. 

Many state coordinators believe the real issues here are whether the 
"programs" are being reviewed early enough in their development and 
whether this approach to developing programs is the most effective and 
appropriate for serving the "needs of the state.",. 

• *■ . ° 

5. Resource Analysis. The word "resource" is used here in the 

broad sense to include all the necessary resources, human and otherwise, 
which are needed to support the proposed program. The analysis of these 
resources is of bo<:h a qualitative and quantitative nature. Some state 
agencies request a complete vitae for each faculty member, while others 
are concerned only with the number of individuals involved. A number of 
states require information on equipment, library resources and facilities. 
Several states require specific information on the administrative re- 
quirements of the proppsed program. 

The - quantitative aspect of this analysis is relatively easy to determine 
and for many factors, such as libraries, states have developed measurement 
formulas. The qualitative aspect on the other hand presents many 
difficulties. Some states use outside peer consultants to kelp in the 
assessment of , the quality of the resources. ... 

A number of states have learned by experience that meeting accreditation 
requirements can be quite costly and that the extent of such costs must 



be taken into consideration when a program is initially proposed. If 
the necessary resources are not available to meet accreditation standards 
most agencies will generally not approve the program. 

» ■* 

Accreditation requirements also are used as indicators of program 
quality. To the extent that a given program currently or potentially 
meets or exceeds the accreditation standards established in a given 
discipline a number of states consider .it to be of sufficient quality 

v J 

for implementation. A full knowledge of accreditation requirements 
can alro provide the reviewers with some notion of the cost to B,e 
incurred in meeting accreditation standards. ■ 

• . v 

6; Program Accreditation. The program accreditation requirement ^ 
can range from a simple statement regarding who, if anyone, accredits 
the proposed program to a detailed listing of the accreditation require- 
ments; the present accreditation status (in the case >f related or 
expanded programs) ; the resources or commitment needed to achieve 
accreditation; and a timetable for meeting the various requirements. 
A few states also require that programs meet general standards of quality 
such as the guidelines of ^he National Graduate Council . 

7. Financial Aid. A few states require evidence of the availa- 
bility of adeauate financial aid for students, "primarily in graduate 
programs. Such requirements usually stem from 1 state policies aimed at 
maintaining accessibility- ... 

The application of all these factors varies not only among the states 
but within a given state depending on the availability of resources, 
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at any given time. v During periods of rapid growth and relatively \ 

\ ■ 

7 

"easy money" the factors may be applied very loosely whereas in times \ 

• \ 

of financial stress the factors may be applied stringently. N 

■* • . 

C. Process of Review 

Different state agencies also use markedly different procedures in their 
reviews of new. programs . The variation ranges from a simple one-step 
process (from institution to' the agency board or commission concerned) 
to elaborate multi-phase arrangements which may. be intimately related 
to the entire statewide "master, ^planning process. For the pu7.poses 
of analysis we will describe nine generally common features of the 
various review processes in a sequence beginning with the initial steps 
and ending with the final steps constituting program approval. 

1. Prior Planning Approval. A number of state agencies require 
that any institution under their purview notify them as soon as initial 
planning for a new program begins. In some states this requirement may 
be fulfilled quite, informally such as by a telephone call or brief 
letter of intent. In Colorado, and in Georgia for example, the institu- 
tions are asked to consult informally with their statewide boards well 
ahead of submitting formal proposals; in Washington and Minnesota the 
notification in advance is required. 

While such expressions of intent are often viewed as bureaucratic (and 

sometimes they are), they can be an important part of the academic planning 

. and coordination process. The following excerpt from a Minnesota document 

explains ore of the major reasons for preliminary ratification: 

The primary purpose of this preliminary notification 
is to provide the Commission with information which 



. it in turn can forward to the various systems of post- 
secondary education in Minnesota. It is >very, possible, 
when a particular educational need develops in the 
state, several institutions of post -secondary education 
may work simultaneously on the planning of new programs 
to meet- the identified need. 

Thus, the preliminary notification can eliminate wasteful duplicate 
planning and encourage cooperation. A number of instances were reported 
in "which the lack of such preliminary information resulted in wasteful 
duplicative efforts in which two or more institutions were simultaneously 
planning similar programs. Furthermore , when the discovery of the dupli- 
cative efforts was made, additional problems often resulted because of 
the institutional hostility created in the process of attempting to 
decide whidfC^if an y> program(s) should be approved. 

Preliminary notification of proposed programs can also feed into the 
statewide planning process and provide the opportunity for state agency- 
institution dialogue. The resulting dialogue between the state agency 
staff and the institutional personnel can. promote the effective use of 
resources even if it is viewed primarily as "building political support" 
fora proposed program at the state level as one dean described it. 
Such interchange can foster greater understanding and a better working 
relationship between agency staff and institutional personnel. It can 
also result in the conservation of resources insofar as a clearer sense 
develops of what is required for approval. Furthermore, it could also 
save time and energy in planning as the state agency staff in one state 
advised that a particular proposed program "had no likelihood of being 
approved at the present time." fi . ■ 

2. ; . Statement of Approval. Tnis second common feature, a statement 
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of- institutional program approval, is often implicit in a request for 
a new program. In some states, however, explicit endorsement by the ' 
legally constituted governing board and/or the institution's or system's 
chief executive officer is specifically required before it can be 
- considered for state-level approval. Where required, it is more of a 
formality to assure appropriate prior approvals and institutional 
commitment to a proposal than anything else. In some states, such. as 
Texas, it is also a' statement attesting to the adequacy of funding for 
' the proposed program. Whether such a statement is requested separately, 
N or as part of the prescribed format required of all new program proposals, 
it is probably a necessary formality in order to maintain proper 
organizational protocol, o 

3, Prescribed Format, Almost all of the state agencies have a 
prescribed format which must be followed by any institution submitting 
a proposal for a new program. This format can be somewhat flexible; 
Texas, for instance, provides only a format to serve as a v general guide 
for institutions submit ting^ requests, _ In most other states, such as 
Wisconsin, rather detailed specific information is required for response 
to a number of items. It has generally been the experience of most 
state agencies that specific detailed formats are the only way of 
assuring that the responses will be something other than vague and 
meaningless. It is admittedly difficult to develop~a~ format requiring ' 
very specific responses which is still flexible enough to accommodate 
new approaches and differing situations, and it is even more difficult 
when state- agency staff are inflexible in their 'determination , to get 
responses to specific formats. If all institutions could be persuaded 
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to respond conscientiously as some consistently do, such rigid formats 
would probably be unnecessary . ' 

«» ■■■*■ 

. 4. Inter-institutional Review. A number, of states include in 

v- ■.. * •\r". 

their review process some kind of review by institutional personnel from 
within the state, usually academic vice presidents, and deans. This inter- 
'institutional review can be informal' as in Washington t and Florida where " 
each institution in. the state is asked to comment in writing on proposed 
programs from similar institutions, or it can be a formal committee such 
as the Council on Chief Academic Officers (COCAO) in Kansas or the 
Curriculum Advisory Committee (CAC) in Minnesota. These conun^.tjtees are 
composed of institutional personnel with state agency staffers serving 
as ex officio members. Whether formal or irifonhal , the iriter-institutional 
committee can be an important source of institutional involvement invthe 
state-level program review process which should not be considered lightly 
by the institutions. Such committees are usually, advisory to, the state, 
board or a subcommittee thereof. , "' 

The most common and recurring problem with ^uch committees is their 
tendency to "log-roll," a process whereih questionable deals or so-called 
*" trade offs" — are.. made. This problem has already been identified by 
other authors; and concerns about it are universal. However, not all 
trade-offs should be viewed as undesirable. Two kinds of "trade-offs" can 
be identified. First, there are thpse trade-offs, in which two institu- . 
tions carve up the turf, agreeing not to develop programs in certain 
areas in return for certain other areas, thus complementing each other's 
activities. This kind of trade-off is generally regarded as desirable. 



On the other handy there are the undesirable trade-offs whereby insti- 
tutions participate in "bloc, voting" . (e.g. , developing institutions 
versus the well developed,, or vice -versa) , or "log-rolling" where one 
institution- agrees\to suppdrt another '$ request in return f of the other 
institution's support of its program requests". In these latter instances 
little if any consideration is given to the; needs of the state or other 
wider objectives of the program review process. In addition to the 
split between the "haves" and the "have nots," one state coordinator 
noted that there are ° | » 

f / - 1 „ i" .i : " ■ ' ; 

related divisions that reflect unresolved tensions within 
the 'academic community over the nature and purposes of " 
graduate. education. These differences, which reflect the 
: differing roles of the institutions in graduate education, 
in turn are reflected, iri^the vigor with which each 
graduate dean reviews and evaluates proposed programs 
according tortraditional criteria. * * 

The "trade-off problem is one o£' the primary criticisms by noneducators 
and state agency staff of the use of inter-institutional committees. 
This and the disinclination of such committees to turn down new program 
proposals contributes toward a general skepticism toward review processes c 
in which primary emphasis is placed on this approach. 

Some' stated Iiave included state agency staffers ex officio to counterbalance 
the problem of "trade-offs." While thi's technique may contribute to better 
state-agency/institutional communications on program review, it probably 
only slightly lessens the probability of trade-offs since such "deals" 
can easily occur outside the formal committee meetings. 

This criticism of inter-institutional committees should not* obscure some 
of their positive-contributions. The concerns about arbitrariness and 
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inflexibility expressed earlier can be counterbalanced by a review 
utilizing institutional personnel with ready access to faculty and 
student concerns . Such committees have also, on occasion, taken on a 
truly statewide perspective by rising above purely institutional self- * 
interests. Admittedly, these instances are rare but they indicate 
that if influential committee members, such as commission members, 
presidents or legislators, can create the proper attitudes in the 
committee,, such committees can be a very important part of the new 
program approval process. Such guidance has predominately been lacking, 
however.' 

It is also important that appropriate procedures and guidelines be set 
up and followed to insure that when institutions submit programs to the 
committee, all prior review work (intra-institutionally) will have been 
of approximately equal quality. The need for consistency is necessitated 
by the wide range of thoroughness and effectiveness of different institu- 
tion's internal review processes. Some institutions and segments we 

\* t. . ... 

interviewed had practically no internal program review while others had 

rather comprehensive review processes. These differences can cause 
difficulties "and unfairness which unfcrtunately often penalize those 
institutions which do a more effective internal review. In one state 
with^several segments, one segment did an especially complete internal 

\ o ■ 

• \ ■ 

revievV for all programs submitted, while another segment undertook almost 
no review of programs. The latter merely passed on to the inter- 
institutional review committee all programs submitted from the individual 
institutions with the idea that if the inter-institutional review 
committee reviewed them. 'favorably, they would then devote staff time 'to 

55 .. ' 



reviewing the proposed programs. The inter-institutional committee 
was thus at a relative disad ge as it reviewed all the programs, 



since few of its members»knew of the qualitative differences in the 
two review systems. It would seem desirable that all of the internal 
processes should be uniform or made known to the committee. 

5. ' Outside Reviews. There are several variations of the use of 
outside reviewers in the new program review process. One variation is 
the "Commission of Scholars' 1 which reviews new programs in Illinois. 
This committee is composed of seven out-of-state scholars national 
reputations for their teaching and research. The "Commission of 
Scholars" is selected by the Board of Higher Education from lists of 
persons nominated by the institutions. The "Commission's" duties are: 



(a) Study areas of critical need for doctoral programs 
to determine at which institutions they should be 
offered and how their initiation and sound devel- 



(b) Review applications by any state university to 
offer a degree program requiring six or more 
years of education or training. In appropriate 
cases the board may act without referral of an 
application to the Commission. 

(c) Evaluate the intrinsic merit of the particular 
proposal . 



(e) Investigate the qualifications of the faculty 
and physical resources of the institution 
proposing the program. 

(f) Conduct such studies and employ, with the 
approval of the board, such consultants as 
are necessary to inform the Commission. 

(g) Make a recommendation to the board. 



A more frequent variation is the use of smaller informal groups or 
individuals to review new program requests in a given discipline. In 



opment may be expedited. 



Determine the need for 6ac 




rogram. 
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some instances these consultants are chosen and hired by the institution 
proposing the program. In vther states the consultants are chosen by 
the state agency staff or the inter-institutional committee. Still 
other states utilize a combination of state-agency and institutional 
selection < of the consultants. 

The use of outside consultants in reviewing new program requests is 
fairly common whether it exists as part of an institution's internal 
review and program development process or as part of the state-level 
program review. The use of consultants was rather highly regarded 
provided that their selection assured reasonable objectivity. Generally 
speaking, objectivity can be improved if there are mechanisms in the 
consultant selection process which provide for joint statewide-institutional 
or inter-institutional selection of the consultants with all segments 
concerned having reasonable .veto powers. IVe will say more about consultant 
selection in the section on the reviews of existing programs. 

The only major criticism made on the use of consultants in reviewing 
new programs was the obvious problem of costs which may be borne by 
either the institutions or state agency. Generally each consultant is 
paid at a per diem rate of $100-$200, in addition to travel expenses. 
Those states which use outside consultants generally find their reviews 

o 

and advice well worth the costs involved. 

We found two instances where there was considerable apprehension over 
the way in which the reports of the consultants were handled. In both 
instances some segment was denied access to the consultants' deliberations 
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(in one state it was the institutions proposing the programs) . A 

more effective approach would allow all parties concerned tc have^access 

to appropriate consultants' reports. 

6. Board/ Commission Staff Review. Like so many other aspects of 
the state-level review process, that involving the state higher education 
staff review varies considerably among the states in terms of the 
extensiveness and quality of the review, and the importance of the 
review in the overall program review process. -A few state agency staffs 
do not review programs at all. In several other states the staff review 
is limited to insuring the institutions' compliance to prescript .1 
formats. In still other states such as New York and Washington, ^the^ 
staff review is the primary component of the review process. In New 
York, the state education department has the final word on proposed 
program's final registration (i.e., approval). In Washington, the 
Council on Postsecondary Education staff makes program recommendations 
to the council. While the staff recommendations are "advisory only" 
the council nearly always accepts them. A typical description of the 
elements of the. staff review process could include the following 
elements: 



(a) Reviewing and circulating as appropriate communica- 
tions of intent to develop new programs, offering 
advice on program submission and development. 

(b) Reviewing new program proposals and circulating them 
as appropriate. 

(c) Reviewing the content of proposed program's documents 
for compliance with prescribed format. 

(d) Verifying the accuracy of data submitted. 
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(e) Serving as liaison in the selection of consultants 
to review the proposed program and reviewing the 
consultants recommendations. 

(f) Reviewing the proposed program for content, costs, 
quality, need, duplication and compliancy with the 
institutional mission and/or statewide plan. 

- (g) Serving as liaison with inter-institutional committees 
and/or board committees reviewing the proposed programj 

(h) Making recommendations to the state agency board. 

(i) Doing follov-ups,. such as post-audit review for 
compliance with conditions prescribed for the 
•programs approval. 

While perhaps "typical" in an average sense, the above activities and 
the extensiveness of those activities varied among the states surveyed. 

The length of time generally allotted the staff reviews ranged anywhere 
from a single day to one or more years. Two to three months was 
generally adequate for the review of most new program proposals. Periods 
9f time of less than two months were often found to be insufficient for 
an effective review and' those of much more than three months tended to 
be- excessive and in some instances so long that they inhibited necessary 
academic vitality and program development. 

. The problems associated with staff review tended to be primarily related 
to the "qualifications" of the agency's staff which was mentioned 
earlier. Regardless of the backgrounds and experience of the state agency 
staff, most institutional personnel felt that the agency staff were 
not the appropriate persons to review academic programs. This was even 
true in a state where the agency staff who reviewed programs by major 
disciplines were on temporary 3-year assignments from those discipline 
areas from the institutions themselves: 



59 



-48- . 

t 

However, this antagonism toward agency staff was not shared by those 
outside the reviewed institutions or by institutional personnel with 
some understanding of the wider context within which institutions within 
a state must operate. State governmental officials who did not approve 
of the staff reviews were rare. Some institutional personnel while 
generally agreeing With their institutional colleagues that no outside 
review would make it easier to initiate new programs, generally saw 
staff program review as either a "necessary evil" or as desirable in 
the long run to maintain the vitality of their own institutions. To 
illustrate, in one state, the faculty senate chairman of a- large research 
institution conceded that while he preferred no outside review, state 
» agency staff review was needed to curb the unnecessary proliferation 
of programs which tends to dilute state resources and inhibit adequate / 
support of existing quality programs. In a number of states institu- ; 

tional executives stated that it was really a matter of alternatives:^ 

/ 

either the state higher education agency staff did the reviews or some S'c 
other less educationally associated group would do them. / 

A number of other concerns with staff reviews were centered on the way 
in which individual agency staff persons operated. Staff were berated 
for wrongly criticizing proposed programs based on incomplete information. 
If the staff had communicated more effectively with the proposing 
institution this kind of situation could have been greatly reduced, 
in one state a new agency staffer's report was highly critical of a 
particular. institution 1 s proposal due to his apparent misunderstanding 
of the purpose the proposed program was intended to serve. This mis- 
understanding could have "been resolved in a matter of minutes had the 
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staffer consulted with the appropriate institutional representatives or 
vice yers&. Instead, the report went to the agency's board where 
considerable animosities were created in a long heated debate over the 
proposal. The result was not only a wasting of the board's time but 
tive loss of credibility for that staffer's work and the furthering of 
*» *dv*isary climate between the state agency and the institution 
.j.r.rcW-dn Unfortunately, some review processes and already existing 
*V<y*srawy % relations combine to work against a free flow of communication 
fest&eon state agencies and institutions , In one state for example", all 
co«i>r,diSications between the agency staff and the institution proposing 
a pvografli had to be through an inter-institutional committee resulting 
i& frequently distorted communications. 

All of these problems with staff review would 'suggest the need for- well 
trained and knowledgeable agency staff, with the experience and 
analytical skills to understand the complexities of statewide coordina- 
tion. The recruitment and training of such people has long been 
recognized as an important problem for state agencies. While training 
programs are being developed by the State Higher Education Executive 
Officers (SHEEO) , the Education Commission of ^States (ECS) and others 
and while beginning salary levels for many state staff have risen, some 
agencies are still badly wanting in this area. The situation in some 
states is further complicated by deliberate institutional actions such as 
lobbying to keep state agency appropriations, staffing, and salaries low. 
In some cases, such lobbying is merely a reflection of the institutions' 
desperate need for resources; in others it is a deliberate attempt to 
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impede the effectiveness of the higher education agencies in the hope 
that the agencies will fail and eventually be eliminated, thus securing 
a victory for institutional autonomy. Ironically, the few state agencies 
that have been "eliminated" have been replaced by more powerful agencies 
that were more regulatory in nature. It appears then that this kind 
of short-sightedness on the part of the institution's personnel can 
often lead to even greater losses of autonomy in the long-run. 

7. Board Committee Reviews. Another approach utilized in a 
number of states is a review by a committee of the board (council or 
commission). This committee may consist of as few as three board members 
or as many as nine or ten. Further variations include some staff and 
institutional memberships. 

Generally, such committees are regarded as helpful in assisting the full 
board in its decision-making and in enabling at least some board members 
to consider in-depth the reviews of new academic programs. This is 
especially true for boards with extremely long meeting agendas. Associated 
with this arrangement, however, are potential problems, evident in several 
states. Some board members, after long periods of service on such a 
committee, become overconfident of their ability to review programs and 
tend to take on an expertise of their own, sometimes substituting their 
own. private views for institutional and board staff judgments on programs. 
In one state a long-time and influential board member on a review 
committee was instrumental in altering the content of a proposed program 
in very damaging ways. The program was saved only by the timely rein- 
statement of its original proposed structure. The institution proposing 
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the program had accepted the board member's changes because it thought 
it might, be the only way the program would get approved. The agency 
staff also did not agree with the changes but found *t awkward to 
oppose them vigorously. 

In general , the full boards tend to support the recommendations of their 
colleagues. Notable exceptions to the acceptance of committee recommen- 
dations were those situations where the recommendations conflicted with 
the wishes of institutions with a particularly close relationship to 
other board members. An instance of this problem was in a state wher^ 
two board members who were not on the review comnittee opposed the 
committee's recommendation to disapprove a proposed program from an 
institution from which they had graduated. 

8. Informal Withdrawal. In many states it is possible for an 
institution to withdraw a program from formal consideration to "save 
face, 11 when it believes that the board Will reject it. This mechanism 
of withdrawal serves several purposes 0 Including the avoidance of 
unnecessary conflict at a public session of a board or commission and 
the option to bring the proposed program back for consideration at a 
more auspicious time. 

9. Review by the State Agency Board. ' In most states the final 
element of the. new program review process is the action of the state 
higher education agency board. In some states the board may review 
new programs only once a year or biennially, associated with the ' 
budgeting cycles. In other states the board 'may review requests for new 
programs at any of its regular meetings. ^ 
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The procedures for presenting the proposed new programs to the board 
vary. A few states let faculty members of the proposing .institution 
make a presentation to the board. In other states the inter-institutional ° 
chairman, board committee chairman or board staff may make the presenta- 
tions. In some instances the consultants may also make presentations to 
the board. In still other states— the presentations may be made by any 
or all of the above groups. 




Developing the appropriate mode for the presentations by various groups 

is important. In several states where the staff has made a negative 

recommendation, inappropriate emotional appeals are sometimes made such 

as the following which was noted by one coordinator: 

.'..there has been a tendency of late for the institution 
affected to have not only those with institution-wide 
responsibilities appear before the [coordinating board] 
but the department chairpersons as well. The result has 
been an emotional personal appeal in which the arguments 

presented sire those which, any good department chairperson • 

or faculty would make to their dean but which do not 
address the issues of greatest concern to 'a statewide 
coordinating board. Unless the staff has done an 
effective job educating [the coordinating board] to the . 
issues and questions involved in the review of academic 
programs, decisions may be made on fundamentally 
inappropriate grounds. Unless there is an active 
process of education of boar^. members to the issues 
involved in program review they do not feel comfortable 
making their own independent judgments and feel forced 
into a position of "taking sides" either-with the 
[coordinating agency] staff or the institution. 



This coordinator also raised the important question of "how, in a public 
forum, does one deal with the question of the qualifications of a faculty 
to offer graduate work?" These concerns raised in a number of states 
indicate a need for guidelines for making presentations to the 
coordinating board and/or for an active educational program for board 
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members that would better equip them to deal with the. issues with which 
they are frequently confronted in reviewing academic programs. 

All of the above approaches had both advantages and disadvantages. The 
effectiveness of a given process seemed primarily based on providing the 
appropriate opportunity for all those affected to express orally or in 
writing tfieir concerns on the proposed program and for those concerns 
to be at least considered in the deliberations of the board.. 

Depending on the authority of a given board the action taken may be to 
approve, disapprove, postpone, recommend, not recommend or a number 
of intermediate actions dependent upon fulfillment of some condition. 
In Oklahoma, for example, approval to implement a program may be 
conditioned upon subsequent funding of the proposed program by the 
legislature. Sometimes approval of a new program is made contingent^ 
upon the elimination of an existing program. Another frequent action 
is to condition permanent program approval upon a satisfactory post- 
audit review to see that the program measures up to the purposes for 
which* it was originally proposed. In Louisiana, for example, all degree 

programs -- associate, baccalaureate, masters, doctorate -- which the 

1 

Board of Regents deems worthy of implementation are initially given 
"conditional approval." In the year that a proposed program will 0 
graduate its first class, a review of the program is required by the 
Louisiana Board of Regents. The reviews in Louisiana include the 
following information: (]) demonstration that requirements of quality 
education are met in the program; (2) evidence that the submitted need 
is met by. the program; (3) evidence that/ the program has served the 
number of students projected in the original application. If a 
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satisfactory review is not bbtained after the first graduation from the 
program, a second review is required upon the second graduation from 
the program. Should this second review prove unsatisfactory to the 
Louisiana Board of Regents, conditional approval is withdrawn and the 
program is terminated. 

Tn North Carolina, the University System requires that proposals for 
new programs include "performance goals for proposed programs in terms 
of projected enrollment, anticipated accreditation, modification of 
facilities and faculty development -- which form the basis for periodical 
review of programs after establishment. 11 ' :s7 ' 
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III. ANALYSIS OF EXISTING PROGRAM REVIEWS 

The review of existing programs is a complex phenomenon. At the present 
time, a variety of states, for a variety of purposes, in a variety of 
ways are undertaking, such reviews. Approximately twenty stats higher 
education agencies currently review at least some existing programs, 
and a number of others are developing procedures to review existing 
programs. 

As noted earlier ctate higher education agency responsibility for review 
of existing programs varies considerably. For example, one state board 
with only the power to approve new programs made an extensive study of 
existing programs apparently with no institutional opposition. Agency 
authority over subject and type of institution also varies. For 
example, only New York and Pennsylvania (teacher education programs 
only) include private institutions in their review of existing programs, 
the oth'er states are primarily limited to reviewing public institutions. 
Some states are further limited in their jurisdiction over the public 
sector. The New Jersey Department of Higher Education has stronger 
review powers over the state colleges than over the other public insti- 
tutions. A similar situation prevails in Pennsylvania where the Board 
of Education has strong powers over state colleges, but has review 
powers over new teacher education programs in both public and private 
institutions. * 

In some cases these reviews are rather limited: they are merely an 
extension of the agencies' review of new programs, such as the post- 
audits described in the previous section, or they are studies of only ■ 
specific kinds of curricula such as the' Minnesota Commission on Higher 
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Education's review of certain health programs offered by public institu 
tions. In other states, such as New York, the reviews of existing 
programs are rather extensive assessments of the quality of graduate 
programs in every public and private higher education institution in 
the state. 

The purposes for which state agencies undertake program review of 
.existing programs also differ widely. Some. of the purposes for under- 
taking a review of existing programs are noted below: 

1. Financial - to cut back on expenditures. One state 
estimated it saved $1.5 million on its reviews of 
existing programs. 

2. Efficiency ~ to enable more effective utilization 
of existing resources. 

3. Accountability - to assure that institutions are 
effectively meeting state goals. , 

4. Quality - to up-grade the overall quality of programs 
by eliminating low quality programs and reallocating 
the resources to improve others. 

5. Consumer Protection - to protect students from 
programs of questionable value. by establishing 
minimum levels of quality. 

6. Political - to demonstrate the clout the agency 
has over higher education. 1 • 

The wide variety of practices in the review of existing programs makes 
generalizations extremely hazardous. What'might look like an excellent 
practice in one state may be questionable practice in another state. . . 
Each state must operate within its own environment.- 

Locating the beginning of a particular state's process of review is 
often difficult, but one which can contribute *to a greater understanding 
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of the total review process for existing programs. While it is not 
, o possible 5 in this volume to trace the background and development of the 
review process, as it is currently practiced in several states, two examples 
of its-evolution can be instructive. Further analysis of the mechanism 
■•for review '.will, follow these examples. * 

. 1. Nfew York . The New York State Education Department has conducted 
two recent reviews of existing programs. In 1969 the department began 
a review of master's degree programs for the purpose of preparing quide- 
liries and 1 making recommendations t^the regents. More recently it has 

reviewed selected doctoral programs. 

' * * * 

Master Degree Review - Intensive planning for the survey began in the 
summer of'1969'with the development of a twenty-one page form designed 
-co reco-i v-;^ chat would reflect che character of New York State's 
master'- degree programs. Fifty-six of the eighty-two institutions 
.offering' a master's degree in the state were selected for the study to 

' r \ 

represent A dross section of the state's institutions. 

During the academic year 1969-70, information was collected by mail on 
enrollments; admissions standards; s'tudent profiles; curricular content 
and development; grading; the credentials, experience, assignments of 
faculty members; and the adequacy' of libraries and other facilities for 
graduate study. Written assessments covering such matters as academic 
advising, library services, the nature of the courses given in different 
curricula, and the quality of instruction were obtained from a sampling 
of students. 

Staff members visited all the institutions included in the project. Two 
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or three persons spent one or two days at each, accumulating additional 
information, "'and gathering opinions from administrators, faculty, and 
students about the nature of the master 1 s degrees offered. A total of 
544 curricula, which included 55,602 courses taken, were examined. 

In addition, twenty-two of the visits were made by teams responsible 
for the registration of college and university curricula. About half 
the members of each team were consultants chosen because of their 
competence in . those fields selected for special attention. /During the 
course of their visits, these teams scrutinized with particular care each 
institution's methods of providing for the administration, staffing, 
and support of undergraduate and graduate curricula in the selected 
fields. Their reports were added to the information derived from other 
sources; then all relevant material was reviewed by the staff of. the 
C/ 1 Bureau of College Evaluation. While no programs were dropped directly 
as a result of this review of Master's programs, the review did Ifsad 

to improved admission procedures and a movement toward performance 

j 

based programs. A report on ithis review was prepared in 1972 entitled 

i 

Master's Degrees in the State! of New York 1969-70 . / 

i 
\ 

Doctoral Education Review - Ii^ January, 1972, the regents announced the 
appointment of a Regents Commission on Doctoral Education which was 
given the charge to make recommendations to the regents for developing 
policy to meet present needs and \to guide the future development -«f 
doctoral education. The commission was chaired by Robbin W. Fleming, 
President of the University of Michigan with the balance of the member- 
ship consisting of noted scholars from around the country. 
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Th e commission had five formal meetings during the year* complemented 
by individual conferences. At two of these meetings, in the spring and 
fall ofM'972, the commission, met with the chief executive" officers and 

. \ 

other representatives of the state's doctoral granting institutions to 
discuss the progress of the study and to solicit their observations. 
In addition, the Regents Advisory Council for Graduate Education, consisting 
of many of the graduate deans of the state's universities, was kept 
informed of the stud>'s Drogress and their observations, too, were sought. 



f institutions in the interest 
ient and economical use of 



The recommendations of the commission in its final report to the regents 
included the following: 

(a) The regents should regard all the doctoral programs at both 
the public and private institutions as constituting together an inter- 
related system for doctoral education. 

(b) The regents should have a general policy of concentrating 
programs at a relatively limited number oi 
of both highest quality and the most effi: 
limited resources. 

(c) The regents should establish special committees to review 
the quality of and need for doctoral programs in selected disciplinary 

-areas. Only programs meeting standards of present or potential high 
quality, and need should be offered. 

(d) New York State should lend its 
public and private sector^ only to programs meeting the standards of 
existing or potential high quality and nepd. Programs without thes< 
qualifications should not be supported. 



r inanclal support in both the 
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(e) New York State should strengthen its support of all programs 
that meet the standards of high quality and need. 

(f) The regents should sponsor increased cooperation and coordina- 
tion in doctora^ education by the institutions within the state. 

(g) The regents should insure that doctoral education at all 
institutions within the state be accessible to all .qualif ied'New York 
students. Economic and cultural barriers to the realization of this 
goal should be eliminated. 

(h) The regents should require that, as part of the 1974. Statewide 
Master Plan Progress Report, all the doctoral-granting institutions be 
required to review their doctoral programs from the point of view of 
determining anew their purpose, place and need in overall institutional 
plans . 

(i) The Commissioner of Education should end the moratorium on 
new doctoral programs when ready to implement criteria and procedures 
that will insure that any new programs fully meet rigorous standards 
of potential quality and need. 

2. Florida . * In 1969, the Board of Regents published a document 
entitled, Comprehensive Development Plan (CODE) of the State University 

System, of Florida, 1969-1980 . The document described "in broad design 

a 

the current operation of the State University System, sets forth in 

general terms the future goals of the system, and outlines the policies 

and procedures to be followed in attaining the stated goals." One 

such ^oal was to provide maximum educationa'l opportunities for the 

citizens of Florida, without unnecessary duplication or proliferation 

*This description is drawn largely from a Florida Board of Regents 
memorandum dated March 8, 1974, entitled "Academic Program Control 
in the State University System of Florida." 
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of programs. Since *:his implied formalized academic program control 
and coordination, the chancellor delegated the responsibility for 
developing necessary implementation procedures to the Vice Chancellor 
for Academic Affairs. 

It is important to recognize that the extent to which academic program 
control^is exercised is directly correlated with the financial resources 
available, student demand, societal needs for individuals with specific 
competencies and skills, the job market, and faculty aspirations. The 
priorities established by a single institution within the system for 
the offering of specific programs may not necessarily coincide with the 
priorities of the system as a whole. Moreover, the legislature takes 
into consideration the many and diverse needs of the entire state, and 
within the framework of available state resources, determines the allo- 
cation to the State University System. Only academic program control 
on a systemwide basis can function with such a statewide perspective. 

In 1969, when CODE was first published, program projections were 
made on the assumption that enrollments would continue to increase in 
future years at the same rate as in past years. But, since 1969 several 
unanticipated situations arose. Although enrollments in the system as 
a whole increased slightly, enrollments in some universities remained 
the same or decreased. Concurrently, the economic conditions of the 
state and nation changed, resulting in fewer jobs available in industry, 
state agencies, and educational institutions for university graduates at 
all levels. The legislature began to -demand accountability for monies 
expended, effort of faculty and staff, the quality of university programs 



73 



and the employability of university graduates. Correspondingly, the 
resources appropriated to the State University System by the legislature 
were less than expected. The University System and its member univer- 
sities were forced to re-evaluate their entire operations, and in some 
casps, to modify their respective lists of priorities. 

Since 1969, a number of controls had been placed on program 
development including: a five-year moratorium on new Ph.D. programs; 
the ini :iation of procedures for identifying duplicative programs; a 
policy requiriug universities to receive authorization tc plan new 
programs; and the submission of formal proposals for new programs. 

In 1972 the Board of Regents took another step, adopting a set of 
academic program control procedures that targeted for further examination, 
programs which failed to meet established degree productivity standards. 
Every year the number of degrees awarded in each program for the previous 
three years is reviewed. If the number of degrees awarded during the 
three-year period under review falls below an established minimum 
number, the program is placed on probation. If .the program remains 
underproductive for three consecutive evaluations, it is then subject 
to an in-depth study, the' results of which will det3rmine whether it 
should be terminated, absorbed as a track in a broader based program 
already authorized, or whether the underproductive program can justifiably 
be continued. 

It would be impossible to conduct a thorough investigation of every 
program every year. The use of degree productivity as a means of 
identifying programs to be evaluated rests on the assumption that with 
the exception of professional programs such as medicine and law, degree 



productivity is the best single index which correlates meaningfully 
with enrollments of majors in the program, student demand, job market 
for graduates and quality of the program. In 1973, 44 graduate 
programs were identified as underproductive, 16 were placed on probation, 
and it was recommended that 22 be absorbed as tracks or subspecialties 
of existing authorized degree programs. The 1974 analysis includes a 
review of baccalaureate as well as graduate degree productivity. On 
the basis of the 1974 analysis, it is anticipated that a total of 113 
degree programs will be placed on probation. Of this number, 9 are 
doctoral, 38 are master's;, and 66 are baccalaureate degree programs. 

Soon after the academic control procedures were implemented, they drew 
harsh comments from around the system. Faculty and academic adminis- 
trative criticism of these procedures focused on several aspects of 
the process: (1) Faculty were most unhappy with a procedure that 
seemed essentially designed to eliminate degree programs ; (2) They 
were also dissatisfied with what appeared to them to be a quantitative 
rather than qualitative approach to the review process. Although the 
regents 1 staff tried on numerous occasions to explain that degree ■ 
productivity was a measure used only to identify programs to be studied 
and that qualitative criteria would be used in all in-depth studies, , 
faculty remained unconvinced and unsupportive; (3) Finally, faculty 
were aggrieved with the term "programs on probation 11 used by the Florida 
press to cite underproductive programs identified by the regents 1 staff. 
They felt programs identified in the public ..nd as "on probation" were 
unduly stigmatized. As could be expected, the term "programs on 



probation" appealed to certain legislators (and perhaps to tire* lay- 
public) who expressed a growing confidence in regential ability to 
"prune" the system. ■ 

At their June 1975 meeting, the regents, responded to faculty concern 

'•**"'■ ...... 

by Directing staff to develop an alternative to current academic 
program control procedures. Degree productivity was not to be the 
only criterion in determining which programs would be studied. A 
set of procedures has since been designed which stresses evaluation 
of existing programs, although discontinuance of low priority, programs 
remains one anticipated outcome. The new procedures are expected to 
be more expensive to operate than the current ones because nonsystem 
consultants will play a major role in the evaluation. All programs 
in the State University System will be c reviewed periodically. Indicators 
will be utilized to identify which program clusters merit immediate 
review. 

The examples of New York and Florida are indicative of a planned, 

rational approach to the establishment of a review process for existing 

programs. In other stages comprehensive icv^ow processes were begun 

or spurred on by economic conditions, such .as the massive reduction of 

a major industry in the state of Washington, or severe fiscal constraints 

such as those suggested by Governor Patrick Lucey in a letter to the 

Board of Regents of the University of Wisconsin in January, 1975, in 

which he stated: 

Specifically, in 1975-77, the University System will 
, need to live substantially within its current year supply 
and expense budget, and find room for additional pro- 
ductivity savings. Budgets for support services, 



computing services, and instructional supplies will 
be less than the level of support previously provided. 
To the extent that larger enrollments come to the 
University System during the next biennium, additional 
enrollment funding may not be available. 

These circumstances are regrettable.* They represent, 
however, the fiscal stringency facing state government 
and the state as a whole. More importantly, they also 
forecast the fact that the University System and the 
state should be planning now to. reduce the scope and/or 
number of its array of higher educational institutions 
and programs in order to be -able to support responsibly 
the needs, of the institutions and programs which are 
to be maintained. 



A. Programs to be Reviewed 

The selection of existing programs to be reviewed varies considerably 
from that of reviews of new or expanded programs. The factors which 
circ?unscribe the choice include legal limitations on agency authority, 
functioMl limitations due to limited staff or resources, and local 
political considerations of how far a given agency should get involved 
in the review of existing programs. These limitations are frequently 
manifested in the reviews being confined to only certain programs 
(e.g., doctoral or all graduate programs); to certain institutions 
(e.g. , public) ; or to certain subject areas (e.g., teacher education*, 
which evidences particular difficulty in placement of graduates). 
Only a few states, such as Kansas, have reviewed undergraduate as 
well as graduate programs. 

Unlike the review of new programs, the actual selection of existing 
programs for review generally involves the use of a screening process 
to determine which individual or similar (lateral) programs should be 
selected for review or, in some cases, selected for more extensive 
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review.* As one agency staffer noted, "Can 300 disciplinary areas 
in the system (not including all degree levels) be evaluated effec- 
tively and efficiently?" While the answer may be yes, most state 
agencies have attempted to screen all the programs in the system by 
the use of a single criterion, and then to review more extensively 
the programs identified as particularly troubled. The University of 
Wisconsin System reviews, for example, distinguish between an audit, 
which is synonymous with the "screening process" described earlier, 
and a "review," which is a full scale evaluation of a particular program 
found wanting in the audit phase. In Wisconsin, all programs are 
audited, but only a few are actually given a full scale review. Great 
care needs to be exercised to assure that the programs selected for 
intensive review are not automatically given a. negative label, such as 
t "on probation," in order to avoid demoralizing those involved with the 
program. This audit and review approach requires as a prerequisite 
an information system which can identify all programs in the system 
by a uniform taxonomy which can provide certain minimum information 
re]|ated to the criteria used to screen all the programs. Increasingly 
th|s program information is being further broken down to include^ info^ 
mation on major and minor tracks within a given degree designation. 

There appears to be a real need in many states for specificity in 
defining programs, concentrations, tracks, majors, minors, subspecialties 
etc. Differences in nomenclature and organizational design make it 
difficult to identify and compare offerings in the same disciplines for 
different institutions. Information on tracks and minors is important 
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because it helps to identify possible areas of program expansion, to 
reiate ^more accurately actual programs with institutional costs, 
resources, and effort and to overcome the problem of definition which 
often results in one institution calling its offering a "program" thus, 
making it subject to review — while another institution calls the 
same offering a f ^track." In addition, information on tracks can be 
valuable as a tool for improving consumer protection. For example, in 
one state -a few institutions with declining enrollments often passed-off 
to unsuspecting students concentrations within a degree program as 
programs. The students' who subsequently enrolled not only found 
limited course offerings, but also found that their degrees had low 
marketability. 

The development of a functioning definition of programs is universally 
frustrated, however, by the lack of a detailed and meaningful taxonomy. 
The most commonly used taxonomy of programs is the Higher Education 
General Information Survey (HEGIS) developed by the National Center for 
Higher Education Management Systems (NCHEMS) for the National Center 
for Education Statistics (NCES) .. One of HEGIS 's shortcomings is its 
lack of compatibility with Handbook VI > the- standard program taxonomy 
for vocational-technical education. Recently, NCHEMS revised the HEGIS 
taxonomy to resolve this particular shortcoming, which was particularly 
distressing for agencies with responsibility for all postsecondary 
education. But, NCES, the primary usev of the HEGIS taxonomy, was 
dissatisfied with the revised version and decided to develop' a whole new 
taxonomy for all of education (K through post-doctoral). 
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In the meajitine states which are attempting -to adopt a common taxonomy 
for academic programs have two choices: they .'can use the original or 
revised HEGIS taxonomy and try to, minimize their shortcomings (not t h e 
least of which is to relate the taxonomy to existing offerings); or 
they can attempt to devise their own taxonomy. The State University 
System of Florida has 'chosen the latter route and is currently imple- 
menting its own taxonomy.- The solution to this problem is not easy and 
must be addressed sooner or later by all° agencies attempting to review 
academic programs... 



5. Criteria Used for Existing Program Review 

The criteria by which existing programs should be reviewed have been 

the focus of considerable anguish for many agencies., A recent Task Force 

on Graduate Education established by the Education Commission of the 

States (ECS) attempted to address this problem by* suggesting that the 

following factors be considered: 

(1) the number of graduates from the program in each of the 
last five years; (2) the number of students enrolled in the 
program (entry and drop-out rates); (3) the size of classes 
and the cost of courses identified as integral elements in 
the program; (4) cost per program graduate; (5) faculty 
workload; (6) program quality as reflected by its regional 
or national reputation, faculty qualifications and level of 
position achieved by graduates of the program; (7) total 
production of a program's graduates from all institutions 
of the state, region and/or natioh; (8) the economies and 
improvements in quality to be achieved by consolidation 
and/or elimination of the program; (9) general, student interest 
and demand trends for the program; and (10) the appropriateness 
. of the program to a changed institutional role or mission. 



An attempt to ascertain the actual use of these factors by the state 
higher education agencies is shown in Figure 2. The table indicates' 
the variation between the ECS factors and the factors utilized by / 
many of the agencies. 
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*Tlic factors indicated for each state are either specifically stated in the material submitted or broadly interpreted to 
be included in these materials. If no items are x for a given state and there are no explanatory footnotes, it indicates 
that no information has been received from that state. 



(1) The criteria indicated has not been approved and is listed here for information only. 

(2) Does not presently have procedures for the review sf existing programs. 

(3) The criteria indicated are for use with new programs and "specifically identified existing programs," Such a 
review of existing programs is based on outside evaluations and criteria developed when the program was proposed 
(if recent). 

(4) Louisiana's coordinating agency was recently reorganized and the materials submitted were based on actions by the 
former agency. It is assumed that" the new agency (Board of Regents) will continue and expand on the earlier efforts. 

(5) New procedures were approved in October 1975, but were not available for inclusion in this report. 

(6) « New. York criteria is based broadly on two criteria "need and quality." §3 
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FIGURE 2 (continued) 



(7) Oregon's criteria consists of: (1) evidence of need for the program; (2) consistency with state goals and 
policies and institutional mission and degree of unnecessary duplication. 

(8) , Information indicated pertains only- to the University of Nebraska system. 

(9) Vermont State Colleges only. 

(10) California has cond-.; >ed studies of high-cost programs in the past, but, does not now have a systematic 0 
procedure for review. 
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The criteria most frequently used by the various agencies were productivity, 
cc ,. quality and need. Productivity was most frequently used to provide 
an initial screening of the programs. Usually, ..productivity was expressed 
in terms of average number of graduates over a certain period of years. 

Some states which have emphasized productivity measures have experienced 
some unexpected negative results, such as a trend toward lowering of 
admission standards and grading practices in programs of low productivity 
.is faculty attempt to save their programs. In one state the heavy 
emphasis on productivity resulted in some programs being later identified 
as "degree mills." A multi-criteria approach was thus identified as 
the most desirable way to review existing programs. One state coordinator 
indicated that it made little sense to drop a program of low productivity 
and leave virtually untouched a program of extraordinarily high cost or 
low quality. Similar arguments can be advanced for all the other criteria, 
including quality. For example, it is hard to justify maintaining a 
program of low demand^ but high quality while sapping resources from. other 
badly needed programs. 

An interesting variation, used in West Virginia, utilizes a system of 

\ 

review which requires forced choices, into three categories for" each of 

\ 

four variables. Siicty percent of the programs are regarded as normal on 
any given variable. Twenty percent are placed on either side of the 
normal range. The variables to be considered are cost, institutional 
priority, quality and output. Each institution has its own method for 
establishing the scale and determining the values assigned to each of 
these variables. Using this procedure each institution selects the 
programs that should be studied in greater depth. 
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C. Processes of Review ' 

The various states', actual processes of program review differ, but they 
can be grouped for the purpose of discussion into four general approaches, 
according to where the brunt of responsibility is located. These four 
approaches include: (1) institutional oriented review processes; (2) 
interinstitutional oriented review processes; (3) agency staff oriented 
review processes; and (4) outside consultant oriented pre *ses. 

1. Institutional Oriented. The institutional oriented approach 
to program review can be of two subtypes. First, many states, such as 
Nevada, have a state-level agency which is not actually involved in 
program review but which encourages the institutions or segments under 
it through various incentives (especially budget) to review their own 
programs. The majority of states probably fall into this category and 
it could be argued that this technically does not constitute a state- 
level review of programs since no one is taking a hard look at the needs 
of the state as a whole. Second, some state-level agencies coordinate 
the individual institutional or segmental reviews in a planning process : . 
which attempts to provide a statewide assessment of needs. Often the 
state agencies using this kind of process provide review guidelines 
and specify dates for their completion. In some cases the agencies may 
even monitor the institutional review processes to achieve quality and 
uniformity. The University of Wisconsin System is perhaps a good example. 
In delegating the task of audit and review to the institutions, the 
University System administrators sought to recognize that the health of 
the academic offerings of each Institution is a primary responsibility 
of its faculty. For this reason, each im citution was asked to develop 
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those procedures which it judj^d most appropriate to its form of governance, 
and most likely to accomplish the goals established. The involvement of 
the central administration in the process is" limited to the distribution 
Qf the general guidelines, the monitoring of the process, the receipt- of 
campus reports, and the dissemination to other campuses of particularly 
effective review practices. 

This approach is probably the least objectionable to the individual 
institution as it allows a great deal of 'institutional individualization: 
the University of Wisconsin at Madison utilised a technique which 
emphasized the use of internal committees,, whereas the Universitv of 
Wisconsin at Milwaukee utilized a technique" which emphasized the use of 
outside consultants. 

Strict adherance to this approach does have the disadvantage of not being 
able to undertake lateral reviews of programs. Thus, if there were an 
unnecessary duplication of a given type of program in the various insti- 
tutions, a strictly institutional review would h, rely on institutional 
initiative to eliminate it. At a minimum the state agencies relying on 
institutional reviews would need to conduct periodic lateral reviews 
and develop mechanisms for balancing the needs of the state. 

2. Inter-institutional Review. An inter-institutional review 
committee is a common component of the review process in a number of 
states. In some instances the committee serves the minimal function of 
assisting the development of the guidelines and criteria.*.. In others, it 
may, in addition,, play an important continuing role in. the review process 
itself. Perhaps typical of the latter approach is that utilized in 
Kansas where the Council of Chief Academic Officers (COCAO) plays a 
continuing role in reviewing, evaluating and coordinating program 
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development at the Regents 1 institutions. COCAO, under the direction of 
the Council of Presidents, also jointly reviews all graduate programs 
on an annual basis. Programs are flagged for further study if the 
average degree productivity falls below cooperatively established > 
minimums. These programs are then singly reviewed and, when warranted, 
are placed on a provisional status. If, at the end of three years, 
productivity has not improved and no exceptional circumstances are 
involved, the program is recommended for discontinuance. The academic 
officer of the Board of Regents serves as an ex officio member of COCAO 
and on occasion may issue through the executive director a separate 
opinion regarding the programs under review. 

The inter-institutional approach to reviewing existing programs has a 
number of advantages not the least of which is the cooperative involvement 
of the institutional staffs. Two states which responded to a Carnegie 
Council survey reported that they "had started out using only staff 
analysis but soon found that institutions cast as passive victims of the 
review would appeal negative recommendations on both procedural and 
substantive grounds, and sometimes won board reversal of staff findings. 

Like inter-institutional committees for new program reviews, described on 
pages 41-44, inter-institutional committees for existing programs reviews 
pose such potential dangers as trade-offs,, logrolling and factionalism. These 
problems can be reduced somewhat by the representation of appropriate stater 
level agency staff and the provision of training for committed members. 

3. Agency Staff Reviews. In some states the agency staff have 
primary or even exclusive responsibility for the review of existing 
programs. In Washington, for example, the staff of the Council on 
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Posts ecohdary Education in Phase I of its review process identified o 
programs, evidencing historical patterns .of low productivity. The 
institutions ^responsible for these programs were then asked to complete 
program review statements and make recommendations to the council 
justifying either continuance or termination*. Specific program . 
recommendations were then developed by the council s.tafr which indicated 
one of four possible actions: program termination; program continuance 
on a provisional basis; the program continuance with further study; or 
program continuance on an unqualified basis. 

Of the 126 programs reviewed, 50 were recommended for - termination, 35' 

were recommended for continued offering on a provisional basis, and 37 

i ' ■' - 

were recommended ror continuation on an unqualified basis. 

A second phase of the Washington review -process is now underway which 
involves a review of duplicate graduate programs, „ 

i " 1 

Staff reviews, if undertaken by senior staff with greajt sensitivity ari. 

judgment, such as those undertaken in Washington received little criticism, 

Howeveij, when such reviews are undertaken by persons of .questionable 

maturity and training they could result in great harm to established 

academic programs and could lead to major confrontations with the 

institutions with long-term negative consequences for both the institu- 

tions and the agency involved. Agency staffers with real- or perceived 

"axes tp grind 11 should not u ~ * ,r en responsibility for program reyiew 

as was [unfortunately "the case in one state we visited. . / ~' 



4. 



/ 
/ 



Consultant c Reviews . A. few- states, in particular New/York and 



Louisiaina, rely on the consultant review process.' In the previous 
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section on the review of new programs we noted the Illinois "Commission 
of Scholars. 11 This group of seven persons, who cannot be residents of - 
Illinois, is also responsible for reviewing (post-audit) six Doctor 
of Arts programs following an experimental period of five years. . 



The consul tan ts<^in New York are selectedas follows: 

1. Names" of distinguished out-of-state scholars* to review 
the ^vograrcs in the state are solicited .from all the 
institutions involved, national professional associations , 
and the Doctoral Council.* A pool of consultants is 
developed from these names. Each academic department .. 
i to be evaluated is -asked to review the'list. and^qross * • 

7 jut anyone who would not be approved as a co nsultant i — ^ 



2. 




Throe of th^ five meirhers of the rating c^m'mittjee in 
each discipline are appointed., by the commf'ssioner from 
the pool of consultants'. Recommendations for nembership 
are made by the Doctoral Council and the invol\ed . 
institutions. This group meets to review all tackground 
material and assigh consultants for. individual ca^us 
visits..- The members also suggest additional Sfecilfic 
questions fox site visitors to particular program^. The 
remaining two members of each rating committee i are 
appointed later from the site visitors. ' j I 

Approved consultantr are then scheduled for cairpus 
visits arid sent the ^c'.gruund material for stuHy. 
The background material is supplied by the iuvolvjed 
institution in response to the questionnaire. \ I 

A team of two or three consultants makes a site visit 
and the chairman prepares a draft report in accordance 
with guidelines provided by the education department. 



the use c of outside consultants is generally regarded as a desirable way. 
o£ reviewing the quality of academic programs. Faculty especially 
feel that peer review is the only meaningful way to review the quality 
of academic programs. Thir is not to say, however, that problems are 
not involved in the use of consultants The fov.r problems we identified 
were: cost , selection," effective use and 'perspective .. 



/ 



*The "Doctoral Council" is composed of graduate deans and academic Vice 
Presidents from various institutions in the state. «• 
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First, problems related to the cost of consultants are obvious — 
if good people are selected, if they all come from out-of-state, and if 
th ,y put in a considerable amount of time, their cost is going to be 
high. The second problem with selection consultants is probably also 
obvious especially to those who have had extensive experience with 
consultants. It is possible to select consultants who will promote 
any conceivable perspective that is sought. For example, in one state 
an institutional representative admitted that he frequently hires 
consultants to review programs from "lesser institutions" who would be 

^^ed by the faculty, resources and facilities and thus write a favorable 

\ 

report^ On the other hand, the opposite result could be obtained from 
hiring consultants from "better institutions" to review programs. 
Needless to say, there needs to be a fair and consistent consultant 
selection process. 

Third, the use to which consultants are put can be problematic. In one 
state consultants were hired to review certain programs but due to 
restrictions on their time, they were forced to do a "motel room study" 
based on data collected by the coordinating agency. They neither could 
visit the campuses nor hear institutional perspectives. As a result 
their report was somewhat discredited. 

Out-of-state consultants can also lack a broader perspective for state- 
wide planning. For example, some New York institutional representatives 
criticized the rating teams for lacking concern for matters of geographic 
access to doctoral programs and for being entrenched traditionalists who 
failed to see a value in serving the needs of part-time urban populations. 
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While the above generalizations are useful for a general discussion of 

j 

the various review mechanisms, it should be noted that it is a rare state 
which entirely fijts into one of these four types. Most states employ 
some combination of the four processes. In order to provide a perspective 
on this mix of types we have developed several matrices (see Figure 3) 
which alqng one dimension snow the various types of roles and on the 
other dimension shows the degree of involvement of selected states. As 
can be seen in these matrices, there is a wide variety of responsibility 
among participants within a state and between constituents in other 

state agencies. - J ^ 

i v 

FIGURE 5 

RESPONSIBILITY FOR REVIEW OF EXISTING PROGRAMS 
Degree of Responsibility Matrix 
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Modest 


Shared 


Primary 


Exclusive 


(1) Institutional 




X 








(2) Consultants 
(outside) 








X 




(5) Agency Staff 






X 






(4) Inter- institutional 




X 
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Washington 



rype of Reviewer 


None 


Modest 


Shared 


Primary 


Exclusive 


(1) Institutional 






X 






(2) Consultants 
(outside) 


X 










(3) Agency Staff 








X 




(4) Inter-institutional 


X 











Kansas 



Type of Reviewer 


None 


Modest 


Shared 


Primary 


Exclusive 


(1) Institutional 




X 








(2) Consultants 




X 








(3) Agency Staff 






X 






(4) Inter-institutional 






X 







Wisconsin 



Type of Reviewer 


None 


Modest 


Shared 


Primary 


Exclusive 


(1) Institutional 








X 




(2) Consultants . 
(outside) 




X 
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(3) Agency Staff 




X 








(4) Inter-institutional 


X 











Florida 



Type of Reviewer 


None";. 


Modest 


Shared 


s Primary 


Exclusive 


(1) Institutional 




X 








(2) Consultants 




X 








(3) Agency Staff . 








X 




(4) Inter-institutional 






X 
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IV. ANALYSIS OF MAJOR PROBLEMS 



A. Costs and Benefits of Program Review 

We.. know of no studies which have carefully examined the costs and benefits 
of program review. In their absence, there is no small amount, of contro- 
versy concerning both the alleged costs of program review and its alleged 
benefits . 



Most state agency staffers are very cautious about giving the impression 
that program review for new and/or existing programs can be easily 
translated into dollar savings. In fact, many assert that program review 
may not result in any savings at all. For example, several programs 
actually terminated as a direct result of the review of existing programs 
were merely paper changes. Frequently, these were low productivity 
programs serviced by faculty in a related area so that the dropping of 
a specific degree designation would result in merely reducing the former 
degree program to a major or track within another broader program. The 
dollar savings, if any, would be quite minimal since no reduction in 
staff or resources allocated would accompany the change. As Lyman Glenny 
has noted: 

Eliminating a program could mean discontinuing a degree in the 
subject and, thus, steering students into some other degree 
program; it could mean discontinuing not only the program but 
all the courses of which the program ir> composed; or it could 
mean dropping faculty members who teach the core courses of 
■She program. 

Of those options, only dropping faculty members, reducing their 
perquisites, or increasing their teaching loads will save, the 
institution substantial amounts of money. If a program is 
eliminated, but n'6t r ?the courses composing it, then faculty 
members must be retained to teach the courses. Moreover, 
if the degree is no longer offered but the courses are kept, 
they may decline in enrollment, increasing the unit costs and 



100 



the costs of the programs for which the courses remain' 
essential. If the courses are eliminated along with the 
program but the faculty members are retained#in some other 
teaching or administrative capacity, no real' savings are 
likely to occur .44 

In some states, especially those in which program reduction was directly 
linked by funding mechanisms to staff and resource reduction, dollar 
savings can be calculated. The Washington State Council has estimated 
that savings of $1.5 million have resulted from its reviews of existing 
programs. The Minnesota Council on Higher Education estimated in 1974 
that its review of new programs altered developments which currently 
"save" the state more than $3 million annually. Other states asserted 
that program reviews resulted in "deferred cost savings," as additional 
resources were not added to programs discontinued or not approved. 

Those considering implementing a program review process with the in- 
tention of saving money and reducing. costs should be aware that program 
review itself costs money. New York estimated that its outside consul- 
tants alone in its review of doctoral programs cost about $30,000 per 
discipline annually. If institutional and agency expenses .were added, 
the total costs would be extensive indeed. 

However, any assessment of cost-benefit ratios in the program review 
process must include a broader definition of benefits than merely the 
funds presumably saved through new program denial or existing program 
elimination. Other benefits are: the institution's establishing a 
high quality program review process and engaging in serious self- 
evaluation; the institution's finding a source of funds for reallocation 
to priority programs in a time of shrinking. resources; the students being 
assured that; program quality is being seriously evaluated; and state 
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officials' and the general public is being macie aware that the institu- 
tions and the statewide board can hold themselves accountable. 



Surely, when these broader possible values are taken into consideration, 

even expensive program review processes ~would seem justified. An 

illustration of the value to institutions of a program review done by 

outside consultants is offered by the comments of a provost of an 

eastern university: 

This is a reflection upon the dynamic rather than static nature 
of the review process as it occurs. As in therapy, which in 
certain respects the review process resembles more than a judi- 
cial process, the review procedure changes the attitudes of 
participants as they go through it. This is true not only 
within each faculty member's own department , but for the campus 
as a whole. It may be observed that within a department, 
for example, there is a predictable anxiety cycle of appre- 
hension before the reviewer's arrival, involvement during the 
visit, relief that things proceed satisfactorily, then mild 
apprehension prior to receipt of the reviewer's report (mild 
because the reviewer is shoWn on campus to be less than an 
ogre, and also because the faculty member has observed certain 
weaknesses in the reviewer which will .allow discounting of, 
any terribly threatening criticisms which may be made) , and 
so on. This is of course understandable. But, more importantly, 
the campus in general is affected by the communal experienced, 
of the reviews, and, as the reviews proceed, there is a tendency 
to become more involved in self-criticism, or at least to become 
more open to the possibility of criticism. This, I would think, 
is quite significant as a precondition for any ongoing self- 
assessment plans. 

Our experience with the reviews encourages me to think that it 
is possible, even in an institution with a strong tradition of 
faculty distrust of administrators , both to have a successful 
series of assessments at a finite point, and to intensify re- 
ceptivity for ongoing sel f -assessment . We achieved a good 
deal immediately. Positions were eliminated and reallocated 
with communal assent. New needs were located} some of which, 
were satisfied in the following year's budget. But even more 
significant was the attempt to create a new mood. Despite, 
the seeming thread to the faculty, the review process can 
ultimately reinforce the faculty by focusing so centrally 
upon it. Seldom is the college faculty member the subject of 
such' intensive professional scrutiny, except during tenure 
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reviews. Rarely does the campus faculty as a whole go through 
a common professional experience. If the process of scrutiny 

is pot entially-^uppox4^ing-^as r: wel.-l -as -pot^ntiaLl-y^-hreat ening-, 

and if the faculty becomes self-critically a participant in 
the process, then the base for continual self-assessment 
should have been established. 

Assuming, then, that the program review process has been judged worth- 
while, what are some typical problems revealed by our research? 

B. Due Process 

First and foremost is the development of a process that genuinely involves 
the institutions. This means more than merely allowed them an opportunity 
to appeal an unfavorable state board decision at the conclusion of the 
review. It ideally includes institutional participation in the very 
design of the process, in the determination of the guidelines and proce- 
dures, in the selection of outside consultants (if any are used) and in 
the deliberations of an inter-institutional advisory committee. 

Such institutional participation is, of course, purchased at a price, 

but the possible delays and informed opposition to. state board efforts 

'< 

are well worth it. Several agencies' initial attempts to get quick 

' * ~> 

program review results by largely by-passing institutional participation 

V ' . . . 

proved to be failures because the institutions presented such effective 
opposition. ' 

It is also important to realize that responding to institutional concern's 
by creating an inter-institutional advisory committee is only the first 
step toward meeting the problem. We have discussed earlier the various 
dangers with runaway committees, log rolling, have-nots ganging up on 
the haves, etc.; particularly in the early months of a new committee, 
a disproportionate investment of senior staff time has to be devoted 
to educating this crucial body about its proper role. On the one hand, 
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the staff must see that centrifugal forces do not come_to_ dominate its 

deliberations; on the other hand, the staff must be hypersensitive to" 
the danger that either in reality or in appearance, the advisory 
committee is not viewed as the "lap dog" of the central staff. . 
Realizing the dangers either way, we urge serious consideration of 
the possibility that some institutional representative and not a 
member of the central staff chair the meeting's. 

Sunshine laws are now opening up many formerly secret deliberative 
processes in state government, so the increasing pressure for performing 
in public- may come as no great surprise. Nevertheless, the program 
review process-- particularly as it is applied to existing programs -- 
is one so sensitive that several states reported, major efforts to , 
reconcile openness of process with sensitivity and discretion. 

An example of this last problem was related by' one state which experienced 
severe legislative pressures for the statewide board to display tough- 
minded program results. A dilemma ar^se when two institutions preferring 
to switch rather than figh^xequested the opportunity to withdraw some 
proposed new programs rather than have them formally rejected in open 
session. While the substantive results would have been the same in 
either case, the state board had to choose whether it was more impor- 
tant to please the legislature or to save the institutions some embarrass- 
ment. 

C. Staff Qualifications 

Statewide boards obviously want qualified staff throughout their agencies, 
but nowhere is the need for it more crucial than in program evaluation. 

104 , 
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—Here-it is-essen\:ial~that-board~slrrff~hrave some" sigriifi cant -prior 

experience in-academe — preferably Including teaching experience. 

/ \ 

The review of new x and particularly of* existing programs necessarily 
* \ 

involves nuanced judgments based on a sound knowledge of academic 

. ' \ ' ''' x 

values. We heard considerable criticism from the institutions that 

in too many instances board staff" lacke^sophlV and made 

questionable recommendations based on simplistic treatment of data. 

t 

A lesser opposite danger we heard only once was that program review 
staff too steeped in academic values and experience might lack adequate 
appreciation of the need to give^the public interest priority over 
institutional interests. 

Staff members working with any inter-institutional advisory committee 
will be under especially heavy pressure; On. the one hand, their 
academic credentials must be impeccable; but on the other, they must 
work conscientiously to see that the committee does not engage in 
trade-offs which protect the institutions instead of the public. 

D. Need for Multiple Criteria, ^Flexibly Applied 

\ 

The mechanistic application of any single criterion in the program review 
process inevitably results in problems. * Most frequently^ an over- • 
emphasis on productivity caused the dilution of ^u^-lity in programs 
identified as possibly underproductive. Lowerpd admissions standards 
and easier grading practices were sometimes resorted to in c order to 
improve a program's numerical productivity. This overemphasis would 
make such questions pertinent: "Does it make much sense to drop a 
good program for low productivity and at the same time leave a program 
bordering on being of 'degree mill' standard untouched?" 
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The National Board on Graduate Education (NBGE) i n its report Doctorate 
Manpower Forecasts ,'and Policy (November, 1973) has also expressed its 



concern over the use of single measures and simplistic formulas in. a 
series of guidelines for states to follow in reviewing programs: 

1. A single measure of quality should not be applied to very 
diverse programs -- programs that may be serving, the needs 
of nontraditional forms of graduate education. Multiple 
indicators of quality, sensibly related to different program 
missions, should be developed. 

2. Statewide planners should resist the temptation to apply 
simplistic formulas to doctoral programs, such as "eliminate 
any program that has not produced more than two doctorates 
within the last two years Such statistical measures may 
flag programs o in need of review, but no program should be 
eliminated on the basis of simple statistics alone, 

5. When evaluating graduate programs, planners should not 
attempt state-by-state labor market analysis, since the 
mobility of the highly educated is certain to confound 
such analyses. A more appropriate criterion, we believe 
is assured access to graduate education for residents 
within the state (or within the region, through reciprocal 
programs)- 45 , ' 

These arguments for multiple measures are further buttressed by a two- 

\ 

year study conducted by the Educational Testing Service, cntitlecj 
Assessing Dimensions of Quality in Doctoral Education: A Techpacal 
Report of a National Study in Three -Fields (1976). • The study noted that 
judgments of quality based upon many indicators had four major advan- 
tages over less comprehensive data. It particularly stressed the 
importance of flexibility, which implies the opportunity to deviate 
from the normal program review process in exceptional circumstances in 
recognition of some special or unique situation. In one state, for 
example, a proposed nonstate funded program was lost due to the absence 
of flexibility in the schedule for reviewing new program ^quests . 
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E. Special Problems iri Measuring Program Quality_ 7 
Most state agencies do not even purport to examine, qualitative factors 
in -their program review process. The Virginia State Council on Higher^- 
Education, -for example, makes a point of noting that its review is a 
quantitative one and that the qualitative review is .the responsibility 

\pf the institutions. Other states, such as Florida, mention quanti- 

■ \ \ ' 

t&tively oriented reviews from which indicators of quality can only be 

gleaned New York's review process on the other hand, has as its major. 

purpose the qualitative assessment of academic programs. 

Quality, however, is an elusive element which is most difficult. to 
measure. Like the question of what makes an artwork beautiful, quality 
is often a relative matter depending upon who is viewing it. The only * 
general agreement we found was that quantitative measures were deemed 
only remotely, if at all, related to quality; and peer judgment was 
the only consistently reliable measure of quality. For these reasons 
the New York State Department of Education chose out-of-state peer 
consultants as a major component of. its qualitative review of academic 
programs . 



But outside consultants are not without their problems. For one thing 
they normally lack an intimate knowledge of inter-institutional rela- 
tionships and the general statewide context. Yet isolated judgments 
of program quality really need to be linked to broader policy, consider- 
ations, which points to the importance of the relations between outside 
coi: ultants and the staff which provides orientation for their efforts 

and of any statewide institutional advisory committee which will be 
• »* 

reviewing their report. ^During the pilot runs of the' New York doctoral 
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program review, therfe were a few comments to the effect thatL-both the 
Jtate Education Department staff and the Doctoral Council reviewing 
the rating committee recommendations were perhaps too much in awe of 
the academic "stars" who bad been used as consultants. As a consequence, j 
one team was allowed to disregard guidelines urgijig attention to 
teaching excellence as well as other factors; it focused exclusively 
on traditional measures of quality: two books published per faculty 
member, faculty depth in most major fields, appropriate library 
facilities and full-time graduate students. This resulted in a. recommen- 
dation to terminate a program at an urban university which claimed that 
it was achieving a statewide goal of broadened' access by serving a 
part-time urban population. 

There were also statewide^ ramifications to an initial;rating team's 
recommendations for termination of programs in other parts of the state, 
a few of which were reversed, some critics skid that,, the original ; 
verdicts showed little concern for matters of geographic access to . 
doctoral programs for students in upstate New. York. . 

Another example of the need to keep the high powered consultants' 

enthusiasms under control was in another reversed' early finding to approve 

/ 

mosfFparts of a doctoral program, but * to recommend phasing out of some ' 
specializations, ^uch' selective fine-tuning was considered excessive 
state intervention and subsequent rating team findings were confined to'. . 
recommendations for departments as a whole. 

One should,' on the other hand, note that there was widespread praise 

"for many aspects of the New York program./ The quality of the consultants 
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was deemed high. The State Education Department's soliciting the help t 
of the institutions to form the 50-60 member consultant panels for each 
subject was appreciated; as was the power given to each chairman of a 
department about to be reviewed to veto from the panel the names of 
persons he considered undesirable as site visitors. Furthermore*' . later 
subject evaluations in New York provided evidence that many of the early 
problems in the use of outside consultants we^re being solved. 

The assessment of quality by state-level agencies may be significantly 
enhanced as a result of the work undertaken by the Educational Testing 
Service (ETS) in the study noted earlier (Clark, Mary Jo, 1976). The 
study clearly demonstrated that the opinions and perceptions of program 
participants and recent graduates were relatively easy to collect and 
use in constructing index sciores for various aspects of graduate programs . 
Furthermore, these measures were found to have reasonably high levels 
of reliability and validity. ETS summarized all the data for each' 
program before comparing programs within each discipline and then used 
the data in several different ways to assess some 30 -aspects of graduate 
programs. From these means, ETS selecte3^23 indicators to profile the 
programs in each discipline. The resisting profiles were then used to 
compare similar departments. 

F. Lie gal Problems 

Although not widespread, legal, challenges to state board program review , 
powers can raise certain difficulties -- if only to delay decisions 
and increase costs. To our knowledge, the only instance thus far in , 
which such a challenge has been mounted is in New York where- the State 
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University took the commissioner of education to court, claiming that 
the authority of the Board of Regents did not extend to terminating 
doctoral programs tnrough a review process. Both the original and the 
appellate state courts, however, have found for tKe regents and their 
powers of program review now seem established. New York regents 
atypical ly also possess program review powers over private higher 
education and have, in fact, acted to terminate. some doctoral programs 
at private institutions. Far from opposing such action, the Commission 
on Independent Colleges and Universities in New York filed a friend 
of the court brief in the above-cited case, supporting the powers of 
the Board of Regents! 

While New York 'State may now be unique in giving its Board of Regents 
such sweeping powers over the private sector, we .can see the possibility 
that pressures will be brought to bear to extend such powers over the 
private sector to -agencies in other states, Appr^^mately^.4_^tates - 
now have programs giving state aid directly or indirectly to private 
institutions within their borders, and as some private institutions in 
deep financial trouble stagger close to the edge, they and their * 
supporters wil] be asking the state for emergency bail-outs. '* It is 
highly likely that public institutions will resent their own programs 
being recommended for termination while related programs of perhaps 
^similar quality at private institutions receive substantial state * 
subsidies and are exempt. Such private institutions had best 'not assume 
that state mpnie:; can be acquired in any significant amounts without t 
"their. surrendering sizable amounts of institutional sovereignty. 



v 

ner^ence of legal is 



Finally, we should mention the possible emergence of legal issues cutting 

/ 

110 



-94- 

across program terminations and collective bargaining in higher education. 
Again, no one has yet drawn our attention to specific instances, but it 
is not hard to anticipate that if program terminations become more^ 
common and result in faculty dismissals or reassignments, negotiating 
agendas will not be far behind. Law suits from faculty unions may then 
become more common against state program reviews than from institutional 
governing boards. This leads us to our last area of analysis. 

G. Political Problems 

While some presidents and trustees are genuinely > piet over what they 
regard as excessive state intrusion into the heart of Academe through 
the program review process, the true level of distress cannot always be 

assessed by public rhetoric. ^Some institutional protest is more "for the 
record" than for* real. A few institutional ^presidents privately admit 
that because of their relations with faculty, students , trustees and 
■i ; m, they are forced to take public positions critical ofr^jprogram 
discontinuation process, whereas in fact they very much welcome some ; ^ 
external leverage to help them crack opien entrenched internal positions. 
It is difficult, then, to interpret just how strong the opposition to 
^s^ate board reviews really is. 

Political pxpblems of another kind arise when influential personalities 
on the board or itl state government subordinate the alleged, rationality 
of the program reviewNprocess to other .considerations of a mere partisan 
nature. Such political intrusions play havoc with state board attempts 
to construct a credible program review process and tend to make institu- 
tions very cynical about • the value of the effort. 
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One example is an inter-institutional committee in reviewing competing 
institutional requests for a new doctoral program labored long and hard 
to come up with an equitable decision on which institution should get 
the program. The committee, after six months, came to a most difficult 
decision. It was an agonizing process that literally almost tore the 
committee apart. When the committee finally made its recommendation 
to the board, the board for entirely "political" reasons related to one 
influential board member's local constituency reversed the decision of 
the committee resulting in its total, demoralization. As one committee 
member put it, "We will never try to do a conscientious job aigain. If 
a' similar situation arises, we will approve both programs even it we 
know that we only need one in the state." 

Another example, comes from'another state in the. I960' s where; log-rolling 
had resulted in a niimbfer of caiiipuses being located in various areas in 
the state. By almost anyone's rational planning, the campuses would not 
have been located where they were, but the situation was semiacceptable 
during a period of rapid growth and almost unlimited allocations, .to 
higher education. The current situation, however, is'^much different: 
appropriations have not been keeping up with the times and there is a 
need to out back on programs. One campus with only a few hundred students 
should by most persons' perceptions be closed; however, legislative 
action prompted by an influential legislator from the affected area 
prevents its closure and program terminations were therefore more 
necessary on other campuses than they really needed to be. 

These examples are illustrative of problems -we encountered in- many states. 
Ironically, some of the self-interested legislators and board members 

7 
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are the ones who loudly scold educators for- not being "efficient. M 
Given the grim prospects for future resources, and the likelihood of 
declining enrollments governmental officials and board members ought 
to practice self-restraint and realize *that postsecondary education . 
effectiveness begins with them. Politics is inevitable in the domain 
of state budgeting for higher education; democrats with a small "d" 
would not have it otherwise. But we urge that partisan politics (in 
contrast to legitimate public policy issues) is not congenial to the 
health of academic programs in higher education, and in our last chapter 
we will try to spall out why we think the statewide board has a urique 
and cincial role to play in persuading the public and state officials 
that the public interest is being served by the program review process. 
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V. RECOMMENDATIONS : WHO SHOULD BELL THE CAT? 

In this'- .chapter we go beyond analysis to prescription, offering as part 
of a necessary on-going debate, our own views on how state-level review 
of academic programs in higher education "ought" jo be done. 

We will first summarize and then elaborate on our principal arguments: 

7 . ■ ''" " - ■■ 

1.. More important than any particular s >t of structures or 

functions is the spirit in which the program review is carried out. 
If it is seen as a process ,which any self-respecting^ institution, even 
a healthy and wealthy one, ought to undertake for purposes of self- 
renewal, rather than as merely- a negative reaction to fiscal austerity 
and/or enrollment; declines , 'then some of the threatening trauma may 

be lessened or even averted. 

•."■■■*. *» 

2. Because such traumas normally involve endangered academic, 
programs,, it. is indispensable, though exceedingly difficult, to involve 
faculty in. the processes. On the other hand, for a variety of reasons 
which will be spelled* out below, a faculty or even an institutional 
perspective by itself is not 1 sufficient . We propose, therefore, a 
state-institutional partnership for review of academic programs. 

3. A variety of actors at the st^te level are potentially relevant 
to program review. We discuss the benefits and shortcomings of different 
officials 1 participation, and then propose a central role for the, state-" 
wide coordinating or governing board, . recognizing, however, the considerable 
difficulties involved in a successful statewide board operation. 
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A. The Purposes of Procrcon Review: Retrenchment or Renewal? 

Even though problems in the process pop up ev n today, the need for 

both institutional and state-level review of proposed new academic 

programs is clearly recognized. After rli, institutional priorities^ 

must be examined; institutional potential for mounting a high 

quality new program must be assessed; state (and perhaps regional and 

national) needs must be evaluated; and a state's ability and willingness 

■v 

to finance new ventures in higher education over and against competing >s . 
priorities must be determined. Sensitive and complex procedures are 
involved, but by and large most states have developed some degree of 
effective cooperation and achieved some measure of success in the 
monitoring process . 

The review of existing programs. is far more problematic, however, with 
few states yet having fashioned a fully satisfactory- set of procedures. 
. Neither the institutional nor the state role has really come into focus. 
Indeed, our exhortation to "think in positive terms". may seem like 
whistling by the graveyard, since undeniably €he process has arisen in 
reaction to state fiscal austerity and/or significant enrollment declines 

First, let u$ establish its legitimacy as a response to these negative 
pressures; then we will try to argue the virtues of this new necessity. 
In both of these arguments, we draw heavily upon the writing of one of 
the more sensitive and articulate practitioners of the art, Donald K. 
Smith, senior vice president for academic affairs, of the University of 

Wisconsin System. ^ v 

' •.. J * • . ■ ■ ■ 

' * 46 

Bowen and. Glenny / have pointed out that the first reaction of most 
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administrators studied in their sample, when 0 faced with state demands 
for immediate cutbacks .in institutional budgets, was to cut across-the- 
board in all programs. This technique had the obvious advantage of 
being both quick and politically less controversial in the short run. 47 
.However, over a longer period of time, most of the systems involved 
converted to some form of selective cuts, recognizing as Smith argues 
that treating all units in the same fashion: 

. . . speaks to an inevitable weakening of the fabric of the 
university as a whole .... the obligation of the univer- 
sity (is) to. search out those programs or activities which, 
'however valued, are less essential to its strength and nature 
than others, and to abandon or alter that which is of lower 
priority to protect the vitality of that which remains. 4 ^ 

Smith is not unaware of the difficulty of moving... a university in this 
direction. .He urges: 

. . the institutional processes of audit and review . ., . 
be perceived as an integral part of institutional mid-range 
and long-range planning, rather than a crisis reaction to 
• ' fiscal stringency. This principle is vital if we are to 
assume that the faculty will internalize responsibility in 
their decision-making for continuous re-examination of 
academic programs. A process aimed simply at program 
excision will never take root in the habit system of 
universities. A process aimed at maintaining the health 
vitality of the institution, whether the fiscal 
ironment is fair or foul, can take root. 49 

Later, we will suggest how mid- and long-range institutional planning can 

be linked and how state planning can complement institutional efforts. 

< 

We would stress, above all, that incentives, not sanctions, be used. 

B. Faculty Participation ^in Program ■.Review: An Indispensable But Not 
. Sufficient Conditio n < 

Again we take a lead from Smith; .. • 

• . . the indispensable condition of a genuine university 
is that- its faculty assume primary responsibility for the 
quality, health, and usefulness of its academic programs. 
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' If this is not done, then the reason for being a community \ 
of scholars with general responsibility for the recovery > 
organization, dissemination, and enhancement of knowledge 5; " * 
is abandoned. Public universities become simply another 
agency of the state or society organized to carry out 
service functions, identified by government or the public 
generally. Agencies external to the faculty may conceivably 
produce quicker decisions on program excision,;^ attenuation or 
alteration than can the faculty, but the price for such a choice 
is inevitably both demoralization of the academic community, 
and the withdrawal of that community into a posture of 
defending all that now exists rather than attending to the 
health and quality of air that exists. 50 

Based on his preliminary experience in Wisconsin, where the governor had 
confronted the University System- with severe fiscal> : VtringericyV Smith was 
optimistic about a faculty's ability to face and make hard > decisions . 
Nevertheless, he found three reasons -- and we can add several more — 

.. . OA 

why 1 even conscientious faculty program assessment may not be enough. 

For one thing, "lateral audits" --, that is, the evaluation of the same 
academic program In all (public) institutions in the state at the same 
time -- are occasionally required. While lateral audits should still 
involve faculty participation, such participation must clearly occur 
in a broader context where the ultimate, authority resides above the 
campus level. While logic suggests the desirability of lateral audits 
in all possible cases, their use conflicts with the competing values 
of local initiative and faculty participation. v 

9 

Unilateral campus actions must also be circumscribed when the recommended 
dismantlement of a given program could have serious statewide or regional 
consequences, or when basic institutional mission, . as set forth in state 
planning, might be affected. For example, an institutional decision to 
terminate' the only School of Forestry in 'the region, or to abandon 
lowerNiivision instruction, would necessarily require a statej-level 
determination. "! 
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Other, more subtle reasons suggest the need for some degree of state-level 
action in review of academic programs. In the first place, the quality ' 
of faculty participation and administrative leadership across institutions 
and systems is very uneven. This means' that, within a given state, the 
statewide board might have three different roles : a light-banded" 
monitoring of institutions which have established high quality procedures; 
a full laying, on of hands for those institutions with no internal procedure 
of their own; and moderate intervention for those institutions which 
have only mediocre program review procedures. 

In the second place, the keeper of the times would suggest a role for 
state-level participation .**^Giyen ' the apparent increase in public 
disillusionment with higher education and the indisputable increase . in 
pressures -for "more public accountability-,. th4 "credibility 11 issue is 
unavoidable: even if institutions are implementing high quality program 
reviews, it is likely that state officials will demand that organs 
^external to r.the system undertake^ separate review process to authenticate 
the internal review. "Justice must not only fee done; it must be seen 
tb be done." 

C. The Search for Objectivity: , Who Should Bell the Cat ? 

Since the focus of this study has been on the role of the statewide 

coordinating or governing agencies, it should come as no" surprise that 

, ' . , a. 

we feel these agoncies ought to play^ a major role in the program review 
process. t But what of other actors "external" to the 'institutions who 
have a role- to- play? . For example, Smith observed in Wisconsin that a 
consortium of four neighboring universities had responsibility £ot audit 
and ^review of graduate programs replicated in more than one of its member 
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UnlV " SitiCS ' ^ ^ « ~* reposed o, any of the four X 
Although the consortium decisions are not le.aily binding Smith reports ' 

th S .r recoraBndattons thus far have proved „ unusuaIly persuas . ve „ 

/ two proMems wit, this Hnd of consorts are obvious: ^ its 

are broader than a si„ gl e institution, the, are stin not statewide, and 
to so,, skeptical state officer eves, consortia. based on voiuntar/insti- 
tutional membership „ in never get rough m ^ ^ 

Some of the same cynicism appUes to two other possibie externa! actors 
whose broader hori.ons answer the concern about statewide dimension, 
both re gi onar accrediti„ g associations and interstate' compact associa- 
tions f such as the Southern Re g iona!: Education Board,, transcend ,tat, 
boundaries; but both-kind, 0 f or g ani Z atio„s depend for their success on 
the g ,neral maintenance- of institutional g ood »„i and support: Thus ■ . 
whiie re gi onai accreditin g associations have been known to «*. harsh ' 
indents about the quality of specific pro g rams, these judgments are 
- nooa::v transmitted privately to the inat^tTonTin question, J and the ~ ~ 

public may never learn of them. Similarlv wh 41 » ,., > • , 

. un '' " hlle the regional compact 

associations may pla y,i leadership role in coordinate the development 
of a special pro g ram area, such as veterinary. medicine in the South 
these organizations are not staffed" to undertake comprehensive pro g ram 
evaluations. Politically, they probably „ oul d not want to assume «, 
role even if they were adequately funded to do ,so. ,■■ . " " 

■ When we turn to the executive and legislative W&MW'f ' 

s * 1 ™.S5j««!^f state government, 
both the canons of objectivity and statewide pii^C^ satufled . 

Their weakness, however, is' in their general lack of famUiarity with * 

nuances of academic -life and values. Particular* in t ' he S ensitaVe area - ' 
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of evaluating the quality \of academic programs is it necessary to know 
the strengths and weaknesse^ of academic peer review, the intervention 
of either bx-and of state government* into higher education presents 



additional problems. The executive branch is preoccupied with state 
budgeting, and the budgeting cycle lends itself very poorly to any 
kind of in-depth program evaluations, especially those in higher 
education where we as yet know so little about how to measure outcomes, 
quantitative and qualitative. Marvin Peterson, John Folger and our- 



selves, in a forthcoming monograph 11 review the efforts of two 
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states to move to some form of performance budgeting, " but there seems 
little in common between such proposed executive staff activities and 
^the kinds of program review we discuss "here . "■ 

- * . '"'■/• 

Legislative program Evaluation, on the other hand,., can embrace intensive, 

..selective program reviews. While few •legislative targets have thus far 

been higher education programs, nothing ...in„ principle, prevents such 

machinery from being aimed at the heart of academe. Even then, if. the 

so-calle'd program evaluation focuses on a systemwide management audit 

v s ■ 

and avoids evaluation of the details of individual academic programs, * 

a complementary division of labor may be achieved in the program' review 
field. Serious questions would arise, however, should state legislative 
auditors begin to . turn their sights on discrete academic programs; legis- 
lative staffs would jnot_seem to ha\?e the appropriate ability or orientation 
to har^Le--tTus7^ problems of overlap with the statewide boards in \ 
higher education would be inevitable. ■ *• 

In theory, -statewide boards are well placed to complement institutional/ 
faculty* efforts, in review of academic programs. In practice, three 
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different problems have occurred. In the first place, some statewide, 
boards may_have to be persuaded to change their operational priorities 
from a preoccupation with quantitative analysis of higher education 
budgets to a deeper involvement with the assessment of program quality, 
particularly as it relates to existing academic programs. Glenny's 
recent massive study of state budgeting for higher education emphasized 
the need for most coordinating boards to shift their priorities from 
\ technical budget reviews and move toward doing more long-range planning, 

program reviews (both new and old) , development of information systems 

■' 52 
and budget formulas, and certain, limited budget evaluations. Glenny 

found that executive and legislative staffs were becoming increasingly' 

, occupied with doing technical budget reviews and were less and less 

willing to defer to- statewide board budget recommendations. While 

some signs indicate executive and legislative staff involvement with 

program evaluation, this development is in its infancy, and the statewide 

boards have advantages herein which should permit tKem a more/ distinctive" 

role . 

/ 

However, a second problem emerges around the issue of institutional 



cooperation. Based on past performances, institutions, instead 



of 



Reusing that they are dealing with the lesser of evils, may actively or^ 
passivelyresist board initiatives in program review; anj^Witl^out such 
cooperation, the board's efforts may fail. 



Finally, if, by some miracle, both the^tatewide/feoafd and the institutions 
in a g^ven state are ready to cooperate, it remains to be seen whether 



in 2L^p*\ 




the program-oriented staff in,the executive and legislative branches 
will , agree to stay out. Clearly, in program issues that relate directly 



to budgeting, they will riot. Hopefully, in matters of academic program 

- assessment,, "chore will be enough credibility in the joint board/institu- ' 

• ■ / 

tional evaluation io convince these staffs not to try to preempt the 

* . / **•? 

field. . / ^ ' . 

v At this point it is necessary to draw some distinctions between the, two 

- majpr Hfypes of statewide boards and the respective strengths s ajid weakness 

they bring to prd^ram review. Statewide governing boards clear lV have 

... • ' — . : '-'"/ 

*■* -the greater power to undertake r'eviews of existing academic programs, 

but practice. to date indicates that they have nd' greater inclination 
to undertake this sensitive task than have their -coordinating agency 
brothers. If anything, the necessarily heavy jmyblvement of consolidated 
boards with institutional budgeting - results ■ in their .having .somewh^n 
less freedom to change priorities in the directions . that Glenny proposes 
h -' and that we endorse. Therefore, on problem number one, i.e., statewide 
' board priorities, we find.po significant advantages to eithfer board 
type. * • > j : '"• cv 

Concerning the second problem, however, this is not so. .Here the 
consolidated governing board is clearly in a better position to "request" 
institutional cooperation, since. the invitation is issued along a * 
governing line of command, with sanctions in reserve if. needed. The. 
coordinating agency, in contrast, must hope for institutional compliance 
but is much less able to enforce it. , , * :\ 



The score is somewhat evened when we come to the third problem, for 1 ' 

' " - \ ' "' v ' "... \. 

statewide coordinating boards are potentially more -credible as "outside 

h ' i ' v ■ r ' ,!V "'"'' 1 - ■ 

evaluators^ than consolidated governing boards, which ave seen; and 

usually rightly so, as "advocates" for the institutions they govern. In 
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a few states where the coordinating boards have allegedly been captured 
by the institutions they are supposed to coordinate, state officers 
dismiss their ability to undertake* "objective evaluations." But in most 
other states, coordinating boards can point to their statewide Juris- 
diction and mandate — similar to that of the executive and legislative -- 
to protect the public interest. 

D. The State/ Institutional Partnership for Quality Program Reviews' - 
Except in those few states where a majority of the institutions already 
have in place a high Equality program review process, we propose that 
the statewide board should initiate a first cycle of rigorous evalua- 
tions of existing academic programs. This cycle would use an inter- 
institutional committee to advise board staff in the development of 
guidelines and procedures, and would exploit board abilities to under- . 

m O 

take lateral reviews and to employ outside consultants. The criteria 
would include the quantitative measures reviewed in earlier chapters 
to serve the audit functidn, and qualitative indices jointly agreed 
upon by the board, outside consultants and the inter-institutional 
committee. 

The statewide board would not make immediate operational decisions or 
recommendations based on this first review, but rather would refer 
its findings to the institutions in question with requests that insti- 
tutional responses be made in the broader and more coherent context of 
the next planning cycle -- preferably every four years. Institutional 
faculty would be involved through participation on the inter-institutional 
committee, through working with the outside evaluation team, and through 
developing proper institutional responses to the findings. 

■ ' ' , , 123.- - . ••: \ 



Maximum use would be made of incentives, thereby institutions would be 
able to retain whatever funds were saved through program terminations 
in order either to enrich other existing programs or to commence limited 
new programs. Other system rewards could be pegge(jt to the planning 
cycle, so that institutions would have positive reasons to respond 
cooperatively. For example, statewide boards could seek a discretionary 
state fund for experimental or innovative institutional projects, and 
, one of the qualifying criteria could be previous cooperation in the 
elimination of existing programs. On the other hand, if an institution 
can offer convincing evidence in the next planning cycle that an 
existing program previously judged weak will receive the necessary 
institutional support, then, absent over-riding statewide concerns, 
the statewide board should defer to institutional priorities. 

The inter-institutional committee should' monitor the fairness of the 
first cycle review, and socn thereafter widely disseminate its mode 
of operations. Institutional self-evaluation procedures should then 
be created and the role of the statewide board should be altered to the 
following three functions: 

1. It should develop with each institution a joint charge for its 
evaluators, and reach a. concensus on the selection of outside evaluators, 
if and when they are needed; 

2. It should monitor the various institutional self-evaluations; 

and 

3. It should be ready tc furnish statewide evaluations should any 
particular institution's procedures break down. 

t 

In order to test the effectiveness of tho monitoring process and to renew 
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its credibility with state government officials, the board perhaps ought 
to select an academic program or two each evaluation cycle to see whether 
a rigorous state-level review produces roughly the same findings as the 
institutional self-evaluation. , 

ff. On-Going Process of Review 

Following the first complete round of audits and subsequent in-depth 

reviews of those programs found wanting, a system of on-going program 

monitoring shpuld be put in place • Many states' have developed infor- 

53 

mation systems which can be adapted for this purpose. . * Also, the 

National Center for Higher Education Management Systems (NCHEMS) has J 

developed a 'State-Level Information Base (SLIB) which can be adapted 

for this purpose. Such a monitoring system might have as its first 

phase a "triggering mechanism' 1 that would be based on program information 

regarding J\e four major groupings of criteria fi.e., productivity, 

54 

quality, cost and unnecessary duplication). Phase two would be a much 
more elaborate evaluation of each of the four criteria indicated above. 

Jr. 

Phases three and four primarily assure "due process/ 1 Below we 
briefly outline how such an on-going monitoring system might work • 

1. Phase One -- Under the proposed process , all existing academic 
^jsffogjrams would be subject to constant review based on an information ,. 
system which feeds data on the four^basic criteria (i.e., duplication, 
cost, quality and productivity). Alternatively, the timing of the 
review cycles could vary; they could be done annually or every 3-4 
years to coincide with the statewide long-range planning periods. The 
cost of the review process would have, to ' be balianced against the benefits 

/ - • 
/ . 
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obtained from a given frequency. The information would be reviewed 

systematically as part of a comprehensive information system. 

\. 

a. Duplication . The criteria used here would need to demonstrate 
whether a particular program "unnecessarily" duplicates another program. 

If so, the program would automatically be subject to a wore extensive 

> 

review, which would have to include whatever other programs it allegedly 
duplicates. A problem would arise, of course, if the ether duplicative 
programs are not within .the purview of the board (i.e., private sector). 
An examination of duplication could be based on the system developed 
by Martorana and Kuhens. 

If duplication proved not to be a problem, the program would then be 
examined on the basis of the second, or cost criteria. If, on the 
other hand, the programs were duplicative, they would trigger a thorough 
going "needs analysis." 

b. Cost . Each program passing the first criteria would next 
be reviewed on the basis of cost. The cost criteria could be established 
as an estimate of maximum costs per program correlated with the level of.. 

programs (i.e., associate, bachelors, masters , 'doctoral) ,~ with years 

\ . 
\ 

(i.e., freshman, sophomore, junior, senior), with discipline or groups of 
disciplines, and other relevant factors. Programs exceeding the estimated 
maximum co:>t wogld trigger a more extensive review to determine if .the 
costs were unreasonable w?iefif other factors were taken into consideration 
(e.g., extreme^iy-'high quality programs, relation to other programs and 
services, etc. ) . 

c. Quality . Quality will probably be the most difficult to 
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assess r and its evaluation may just consist of negative criteria. For 
example., the following might constitute the criteria against which the , 
programs are examined: (1) an unfavorable accreditation report; (2) 
fpj! low-up studies which indicate an unusually high failure rate for 
graduates to pass qualifying exams or obtain future relevant placements; 
(3) low professional standing on the basis of national rankings ox the 
determination of outside consultants. 

A system for assessing the dimensions of quality could also be based on 
program profiles such„as those developed by Clark, Hartnett and Baird;^ 
or the check-list in Frederick E. Balderston.^ 

If a given program is identified as possibly being low in quality, it 
would then be subject to a more extensive qualitative analysis. 

d ;? . Productivity . Minimal standards of productivity could be 

tJ -' ' • ■ ■ - 

stated as an average of so many graduates, at the different degree levels 
over a. given period of time (e.g., 3-5 years). Programs falling below 
the established minimums would trigger a more extensive review of 
productivity, those which are above the minimum would be subject to no 
further review during this review period. 

2. Phase, Two — The second phase of the review process would be a 
more extensive review of the programs identified as troubled in Phase One 
The purpose of this phase would be to insure that the program is, indeed a 
wanting criteria. It is included to prevent ian arbitrary and rigid 
application of a single criteria. 

a. Cost Analysis. It is possible. that *n excessively high 
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cost may be justified on the basis of "other unspecified criteria, which 
would have to be justified as- acceptable alternatives. The burden of 
justification would be on the institution or institutional unit (i.e.,/ 

department, college, division). It may also be possible that a cost-/ 

r 

benefit analysis might be the most appropriate cost criteria for a given 

I 

program. If on the basis of the Phase Two review the program costs 

cannot be justified the program would be submitted" to a need analysis. 

':/ 1 
U 

. b. Qualitative Analysis . Programs identified as being possibly 
of low quality would be'subject to an extensive qualitative analysis. 
Such an analysis would be conducted by. other faculty and appropriate 
administrative' personnel at the institution in question and/or by 
appropriate outside consultants. The review might include a thorough , 
going review of faculty qualification and activities, and of the curricu- 
lum. Programs failing this review would also be subject' to a need 
analysis. . \ c . . 

c. Productivity Analysis .\p£'ike, a the other Phase Two analyses : 

: . \ ' - • . • 

this one would seek to justify (if possible! the productivity of a given 

'. ■■" * • * . ^ ; .. ' 

program which, did not meet the minimum st£(ndards adopted in Phase One. 

The low productivity mighr., for example, be 'justified for a given 

program because it is a necessary component 'of another, high productivity 

program. At any. rate, programs . unable to justify their low productivity 

would be subject to an extensive need analysis.. 

V 

ft 3. Phase Three -- 

. / - > . • * 

a. Need Analysis , This phase, a thorough examination of the 
need for the program^ is crucial in the program review process. Through 

I ' • ■ " ' 
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need analysis programs of questionable cost, quality or productivity 
" are either justified for continuance on the basis of "other redeeming 
value" or the/ are terminated (or recommended for ^termination) . It.' 
could be that the program in question fulfills the institutional 
mission, or supplies state, regional or local manpower needs, or answers 
to student demand or political necessity. > 

The programs that are justified woujd ^likely fall into two categories : 
(1) justified to continue without conditions, and (2) justified 5 to 
continue with conditions. According ' L to the program's particular short- 
coming, its status would be conditional on its improving its quality, \ 
its productivity, or. its costliness. Likely, these improvements; would 
have to be made within a specified time. 

4. Phase Four This last" step in our hypothetical process of 
program reviiew would be the phase in which programs' compliance f with 
the conditiqiis, timetables and other evaluative factors set fdr them 
would be examined. Those- programs failing comply would be terminated 
or recommended for termination. The next step in the review cycle is 
.to return to Phase One. 
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